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ABSTRACT 

The new "Mathematics Framework for California Public' 
Schools" identifies complex instructional activities and student 
performance, and requires new assessment methods to reveal that 
performance. One of the goals of the California Assessment 
Collaborative (CAC) is to study the effects of local assessment 
development and implementation efforts on teaching and learning. This 
report describes how one local CAC mathematics assessment initiative 
connects with the statewide curriculum reform and impacts what occurs 
in the classroom. The Mt. Diablo Curriculum-Based Alternative 
Assessment of Mathematics (CBAAM) was the program examined through 
interviews of 59 third-grade teachers. Their responses made it clear 
that the CBAAM had profou id short-term effects on teaching and 
learning. Teachers resequenced content, introduced new content, and 
emphasized instructional processes as a result of the CBAAM. Evidence 
also suggests that there will be substantial long-term impact. Two 
tables present study findings. Two appendixes supplement the report 
with a student booklet from the 1994 Curriculum-Based Alternative 
Assessment of Math, the third grade survey letter, and interview 
guiding questions. (Contains 18 references.) (SLD) 
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Teacher Perspectives of the Impact and Validity of the 
Mt Diablo Third Grade 
Curriculum-Based Alternative Assessment of Mathematics (CBAAM) 

I. Introduction 

In California, a major component of educational reform involves linking both state and local 
assessment to new curriculum frameworks. The new Mathematics Framework for California 
Public Schools (CDE, 1992) identifies complex instructional activities and student 
performance, and requires new assessment methods to reveal that performance. Although 
the demands are present and many districts are initiating assessment reforms, there is a 
paucity of research and information on how newly developed assessment strategies impact 
teaching and learning in classrooms. One thing is certain; any real impact of the present 
wave of reforms will be mediated through the knowledge and actions of teachers. Therefore, 
identifying the effects of teacher participation in assessment development and the 
assessment s impact on teachers* day-to-day practice is a timely and informative venture. 

Since 1991, Far West Laboratory H: Educational Research and Development (FWL) has 
partnered with the California Assessment Collaborative (CAC) to study the effects of local 
assessment development and implementation efforts on teaching and learning. The CAC 
was designed to systematically identify, validate, and disseminate alternatives to 
standardized testing throughout the state. The initial phase of the CACs research involved 
synthesizing information from many diverse assessment initiatives and developing a 
conceptual model portraying the many components necessary for the design of 
"instuctionally sound assessments." Five key dimensions identified and elaborated by the 
conceptual model include: articulating content standards, developing meaningful and fair 
assessments, developing teacher capacity to use assessment information to improve 
instruction, developing student capacity to use assessment to improve learning, and 
determining and monitoring consequences (Jamentz, 1993). 

The research reported herein builds on the early work of the CAC and places that work in 
the context of the statewide mathematics reform effort, This report describes how one local 



CAC mathematics assessment initiative connects with the statewide curriculum reform and 
impacts what occurs in classrooms. The research was designed, in part, to help the school 
district understand the impact of the assessment reform, and to provide information that 
could assist the district with decisions relevant to the future course of the project. In this 
context, the Mathematics Framework, corresponding state assessments, and local assessments 
represent a system in transformation. The system will only work for students to the extent 
that sound reform strategies play out in their schools and classrooms. Local commitment 
converging on a coherent program of curriculum development, teacher professional 
development, and instructionally sound assessment is the linchpin of the larger mathematics 
reform effort. In this study, the Mt Diablo Curriculum-Based Alternative Assessment of 
Mathematics (the CBAAM) initiative was the point of entry. However, based on the early 
work of the CAC, it was anticipated that this study would come full-circle generating 
substantial information on the nature of classroom instruction and the professional 
development supporting its implementation. Thus, data collection and interpretation was 
guided bv the following clusters of research questions: 

1) Wfiat is the impact of the CBAAM on the mathematics instruction delivered by third grade 
teachers? How does knowledge of the assessment content alter the patterns of instruction? 
Do teachers use the CBAAM results? //so, how do they use the assessment results as an 
instructionaVtool? 

2) Does the CBAAM contribute to teacher professional development? Does it help teachers 
articulate ideas forwarded by the 1992 Frameworks? Does it illustrate and communicate 
important ideas? Does the CBAAM implementation, or use of the CBAAM results foster 
within and across grade articulation of mathematics programs? Are there systematic 
differences in the opinions and actions of teachers involved in the CBAAM activities versus 
those not participating? 

3) Wfiat do teachers think about the development process and the quality of the CBAAM itseip 
Do teachers think the CBAAM is instructionally sound? If not, where does it fall short? 
VV/wf changes do they recommend? 
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n. Background 



Curriculum and Instruction in Mathematics 

This section reviews recent trends in mathematics assessment and instruction and identifies 
several forces that have motivated change. The alignment of assessment and instruction, 
reseaich in cognitive psychology, instructional guidance in the form of standards and 
frameworks, and the critical role of teacher professional development are discussed. 

Mathematics Assessment and Curriculum. The shift from externally produced assessment 
procedures to locally developed performance-based assessments reflects a major departure 
from traditional practice in many school districts. Historically, the alignment between 
traditional tests and the mathematics curriculum has been quite good. Analysis of curriculum 
and assessments reveal that the current mathematics content being covered is limited. 
Reviews of both criterion-referenced and norm-referenced tests reveal items that focus on: 1) 
basic procedures for computation; 2) traditional word problems that involve recognition and 
direct application of basic algorithms (Madaus, West, Harmon, Lomax, & Viator, 1992); and 
3) the recognition of various shapes, quantities, and measures. Likewise, mathematics 
programs throughout the country have focused on a limited range of student performance. 
Gehrke, Knapp, and Sirotnik (1992), graphically describe this rather bleak scenario. In 
contrast to other subjects, mathematics instruction exhibits a narrow range of instructional 
formots where lecture, individual seatwork, applying the right algorithm, and getting the 
right answer dominate. Instruction emphasizes procedural skills, focuses on a narrow range 
of topics, and repeats material in an increasingly redundant fashion as students move 
through the grades. Little attention is given to conceptual understanding, complex problem 
solving, or real-world applications. Additionally, studies have found that mathematics gets 
much less attention compared to reading and language arts in the elementary program 
(about 40 minutes a day compared to 150 minutes in reading and language instruction) and 
that most of this time is spen; on basic arithmetic (Bell, 1974; Porter, 1989; Rowan, & Guthrie, 
1988 ). In general children are not exposed to the range and depth of mathematical ideas 
necessary to develop the depth of understanding forwarded in the frameworks. Mathematics 
curriculum and instruction is changing, but significant change takes considerable time and 
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effort. Teachers are not only changing how they assess their students, but altering the ways 
in which their students come to know mathematics content. 

A Perspect ive from Cognitive Psychology . A major component of the reform in mathematics 
instruction involves viewing learning, curriculum, and assessment from a cognitive 
perspective. In the past, both mathematics curriculum and assessment focused on mastery of 
basic routine skills. This state of affairs was consonant with early learning theories that 
suggested cc iplex, higher-order skills were acquired bit-by-bit by chaining prerequisite 
knowledge. However, during the past decade, research in cognitive psychology indicates 
that all learning, particularly complex reasoning and understanding, is the result of active 
construction by the learner (Shepard, 1992). in fact, even the most basic skills requiring 
memorization and recall of facts involve mental elaboration and judgement (Resnick & 
Resnick, 1989). Resnick and Resnick conjecture that often students fail to recognize the 
connections between formed rules taught in school and their own mathematical intuitions. 
Students think of school math as something different and do not integrate this with their 
practical knowledge of the real world. Often, math is taught, and students are asked to 
demonstrate mastery, in decontextualized situations. The students are then left with the 
difficult task of recontextualizing and integrating this information intc their own knowledge 
base. New approaches to instruction and assessment are designed to connect mathematical 
knowledge to students' daily experience. This places enormous demands on teachers and 
educational institutions to enhance their own capacity to design and implement new 
approaches to mathematics instruction and assessment. 

Standards and Frameworks. There is significant interest in redirecting what has been taught 
and assessed to what should be taught and assessed. Much of this work has been inspired by 
the National Council of Teachers of Mathematics (NCTM). The Curricul im and Evaluation 
Standards for School Mathematics (NCTM, 1989) reflect a national consensus on what 
curriculum and assessment in mathematics should look like. The standards emphasize broad 
coverage of mathematical ideas, the conceptual underpinnings of tools and procedures, 
application of mathematical ideas to real-world situations, and instruction that encourages 
learners to formulate multiple solutions to complex problems. The NCTM standards provide 
compelling arguments for the integration of group collaborative work, open-ended tasks and 
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investigations that may take hours or days to complete, and the use of manipulatives and 
multiple representations of mathematical ideas. Implementing these new standards and 
corresponding assessments will be difficult because they represent a departure from past and 
current practice in many schools. 

MithcmitiCll PQWCr, The revised MathemaHra Framework for California Public Sr hnnl s 
(CDE, 1992) addresses many of the challenges presented in the NCTM standards and builds 
on and extends the ideas in the 1985 framework. This new framework has as its central goal 
the development of mathematical power for all students - "the ability to discern mathematical 
relationships, reason logically, and use mathematical techniques effectively" (Mathematics 
Framework, 1992, p.3). The document describes four dimensions of mathematical power, 
including mathematical thinking (e.g., classifying, comparing, inferring, verifying); 
communication (coherent expression of thoughts and results); mathematical ideas (content, 
e.g., counting, addition, geometry); and tools and techniques (e.g., computational algorithms, 
visual organizers, manipulatives, calculators); Students employ these four dimensions of 
mathematical power to perform work that is purposeful and personally meaningful. 
Instruction and assessment focus on the broad demand for mathematical power through 
coherent units that emphasize large pieces of work. Redesign of assessment practices at state 
and local le* 'els are necessary to capture these complex student performances. 

Professional Development. Research and experience with past reforms indicate that most 
efforts to fundamentally alter teaching and schooling have had modest impact, at best. 
Changes usually consist of piecemeal adoption of new practices, or the translation of new 
ideas into old practices (Cohen & Spillane, 1992). There is also a sense that most elementary 
teacher* lack a deep understanding of mathematics themselves, compounding the difficulty 
of improving curriculum and instructional practice. Altho ugh most elementary teachers must 
. teach mathematics, few feel well qualified because of a teacher preparation system that 
perpetuates a secondary role for mathematics and instructional time allocations that are 
grossly inadequate (Bell, 1974; Gehrke, Knapp, and Sirotnik, 1992; Porter, 1989). Although 
guidance for systemic improvement in the form of mathematics standards and frameworks 
has been devised and widely disseminated (e.g., Mathematics Framework, California 
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Department of Education, 1985, 1992), altering day-to-day practice remains the major 
challenge. In California, changes in classrooms h?ve been slow, and what we now 
understand is that teachers must be extensively involved in building frameworks, curricula, 
and assessments in order to enhance their capacity to implement them. Without this 
commitment to staff development, the guidance offered in these frameworks will not be 
translated into practice (Cohen, et al, 1990). For teachers and school districts, assessment 
development projects represent substantial professional development opportunities. 
Assessment design activities contribute to the articulation of new ideas and standards 
forwarded in the frameworks, and increase teachers capacity to develop and use new 
instructional strategies. 

Meaningful changes in local assessment strategies must occur concomitant with alterations in 
the pattern of classroom instruction. Changing the mosaic of mathematics instruction and 
assessment involves an ongoing commitment to a long-term program of staff development, 
communication, and collaboration. The meaning, validity, and impact of both inshuction and 
the assessments that track its effects depend critically on the nature and quality of local 
support and professional development initiatives. The Mt. Diablo CBAAM represents an 
assessment development program in its fourth year, that relies on teachers to develop and 
implement an annual district-wide assessment. This presented a unique opportunity to study 
the impact of a well established performance-based assessment reform on the curriculum, the 
instruction, ar 4 the professional development culture in a district. The challenge for the 
district is twofold; creating new and challenging assessments while simultaneously altering 
the contei K and process of day-to-day instruction. 

The CBAAM Description 

The -BAAM is one of 29 pilot projects in the CA'C. For four years, the CBAAM has been 
developed, administered, and scored by teachers. Approximately, 5,000 students in grades 
three and six are assessed annually. The CBAAM in an on-demand assessment comprised of 
three constructed-response items. Due to limitations in resources and a large sample of 
participating teachers, this research focused on the impact of the third grade assessment only. 
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Purposes of the CBAAM When reviewing documents related to the CBAAM, four major 
purposes are identified by the school district (Nicoll et.al., 1994). The first involves an 
attempt by the district to bring their assessment system in line with current thinking on 
student assessment and to provide measures of "higher order expectations" that require 
students to generate rather than select answers. Students are asked to solve complex 
open-ended problems and then describe their approach in writing supported by the use of 
appt iate mathematical tools. The second purpose is to engage students and teachers in 
what may be their first attempt at an open-ended performance-based assessment. They 
intend this to be a positive experience; one that students and teachers can use to prepare for 
similar assessments in future years. Third, the CBAAM is designed to prepare students and 
teachers for similar problems on the California Learning Assessment System (CLAS) 
administered in grades 4 and 8. It encourages articulation of mathematics content and 
instructional strategies within and across grade levels by asking teachers to engage in the 
instruction that prepares students for the new expectations. CBAAM represents a formal 
occasion for teachers and students to engage in the requisite actions necessary for success on 
these new and different types of assessments. Finally, the CBAAM is one avenue for 
introducing parents to the ideas forwarded in the current mathematics reforms and the 
related assessment techniques. 

Problem Development Tasks for the 1994 CBAAM were developed by a committee of 12 
third grade teachers and two central office staff. They began with the following guiding 
questions: 1) Can the child express mathematical thinking, ideas, and, processes in written 
and pictorial form? 2) Does the child use a "reasonable" strategy to solve problems? and 3) 
Does the child apply concepts related to the unifying ideas (How many? How much? 
Finding, making and describing patterns, and representing quantities and shapes) to solve 
problems? 

The committee generated a number of potential tasks, then field tested and refined the tasks 
during the fall of 1993. Three tasks were selected from a larger pool to appear on the 
assessment (These tasks and the student response form appear in Appendix A). On 
November 30 and December 1, 1993 third grade teachers participated in a three hour 
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in-service that reviewed the content of the CBAAM, administration procedures, and provided 
suggested classroom activities related to the content. 

Administra tion and Scoring. The CBAAM was administered between January 10 and 
January 21, 1994. Teachers administered the assessment to their own students. According to 
teacher responses the assessment took between three and six hours of class time to complete. 
Teachers had administration guidelines that allowed for flexibility in student groupings and 
the time allowed to complete tasks. Basically, students were allowed as much time as they 
needed to complete the tasks. Tasks were initiated with group activities. However, each 
student produced individual responses to all tasks. In past years, the CBAAM was 
administered in the spring. The decision to go with a January administration was based on a 
teacher and district desire to receive the results earlier in order to use them in subsequent 
planning and instruction with the children. 

The CBAAM is scored by Mt. Diablo teachers using six-point holistic rubrics. A team of 
third grade teachers and district staff refined the general CLAS rubric (See Appendix A) into 
a more detailed rubric for each of the CBAAM items. Teachers were paid a stipend to score 
papers and teachers from all grade levels participated. Teachers participated in a training 
session that instructed them on the scoring procedures and proper use of the rubrics. 
Scoring took place on two Saturdays; January 29 and February 5. All papers were double 
scored with the first rating recorded in ' invisible ink." Papers with a score discrepancy of 
more than 1 point were moderated by an "experienced" teacher panel. The district estimates 
that less than 10% required moderation (Nicoll, 1994; personal communication). Following 
the ratings, papers were returned to individual teachers and students with double-rated item 
scores (e.g., 3/3, 4/5). 

Reporting. Results are aggregated district-wide and published in a report to teachers and 
district staff (Nicoll et.al., 1994). Tables for each item (See Appendix A) illustrate the 
percentage of students scoring at each performance level (1-6). Samples of student work at 
three levels (2,4,6) are also included in the district publication. Score breakdowns by school 
or teacher are not published. Scores aggregated by teacher within school are only provided 
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upon request of the building principal. Historically, few principals have requested this 
information. 
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III. Method 



Site Description 

Mt. Diablo Unified School District is a large suburban district serving approximately 34,000 
students in elementary, intermediate, and high school programs at over 40 building sites. In 
the past four years, the CBAAM has been administered in all third grade classrooms at 26 
elementary schools. Each year, approximately 2,500 third grade students are administered 
the assessment by their classroom teachers. There are substantial variations in student 
populations at the school-sites as evidenced by poverty counts ranging from 1% to 70% and 
limited English proficiency (LEP) counts ranging from 0% to approximately 40%. Elementary 
school populations range from 450 to 850 with a typical building having three or four third- 
grade classrooms. The Mt. Diablo site and teachers participating in this research represent an 
opportunity sample through which to study the impact of a district developed assessment 
that has been in place for several years. All third grade teachers were asked to participate in 
the study. 

Research Sequence 

The first stage of the research involved participation in assessment development meetings 
where items were refined and discussed and materials and communications for all teachers 
were prepared. This allowed, for a review of materials, related publications, and procedures 
employed as part of the assessment program. Following the CBAAM administration and 
scoring, FWL constructed and disseminated a survey to all third grade teachers. The final 
stage involved individual interviews of a sample of teachers. 

Survey Procedure 

Survay Development. The survey was developed to serve three purposes. The first involved 
obtaining background information about teaching experience, formal subject matter 
preparation in mathematics, related professional development and inservice preparation, and 
the level of involvement in the CBAAM development and implementation (See Appendix B 
for complete survey). This information served as the basis for group comparisons of the 
surv ey data. It was also used to select a target group for individual interviews. 
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The second and most vital component of the survey asked teachers to rate 58 items on: 1) the 
frequency with which students engaged in a variety of mathematics classroom experiences 
and were taught specific content linked to the CBAAM items (20 items); 2) teacher opinions 
about their own ins true, on, professional development and the CBAAM preparation (23 
items); and 3) teacher use of classroom assessment strategies (15 items). Each item consisted 
of a statement followed by a five point Likert-type scale. The survey items were developed 
based on the content and instructional practices forwarded in the Mathematics Framework 
for California Public Schools (CDE, 1992), the 1989 NCTM Curriculum and Evaluation 
Standards For School Mathematics (NCTM, 1989), and a review of content and process 
requirements of the CBAAM items (e.g., group work, use of manipulatives, understanding 
coordinates and coordinate systems). A few items wero included that were inconsistent with 
these guidance documents to serve as checks for positive response bias. 

The third and final section of the survey consisted of an open-ended request for comments 
about the CBAAM and an invitation to participate in an individual follow-up interview. 
These teacher comments were valuable additions to the interview data because they included 
responses from teachers that Were opposed to the CBAAM, but elected not to participate in 
an interview. The survey content was reviewed and edited by four CAC and FWL staff 
familiar with the CBAAM project and two Mt Diablo staff responsible for coordinating the 
CBAAM development. 

Survey Participants. Ninety teachers were included in the pool for the survey component. 
All regular third grade teachers assigned to self-contained classrooms were included. 
Specialists, Chapter 1 teachers, and special education teachers were not targeted in this study. 
Fifty-nine surveys were returned representing a 66% return rate. All teachets had experience 
teaching multiple subjects (reading, language arts, math, social sciences) in the elementary 
grades. Teacher experience ranged from two to 32 years with a mean of 16.1. Few teachers 
reported formal mathematics preparation beyond college algebra and mathematics /teaching 
methods courses. Most teachers reported professional development in mathematics 
education beyond teacher training or college programs. These included, subject maUcr 
projects, EQUALS-Lawrence Hall of Science, Marilyn Burns Math Solutions, and several iocal 
and regional mathematics inservice programs. 
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Survey Distribution. Surveys were produced by FWL and disseminated via the director of 
evaluation for the Mt. Diablo School District. Teacher surveys were sent to building 
principals with a cover letter describing their purpose and expected return date. Principals 
placed the surveys in teacher mailboxes at each school-site. Individual teacher survey 
packets included a cover letter to teachers, the survey document and a stamped/self- 
addressed return envelope. The surveys were distributed during the first two weeks of 
February 1994, following administration of the CBAAM. Most were returned immediately, 
with a small number trailing-in through late March. Surveys were returned from at least 18 
of the 26 schools. Many completed surveys were returned without teacher or school 
identification. 

Survey Analysis. Of the 59 surveys returned, it) were eliminated from the analysis due to 
large blocks of missing data. In some cases teachers did not fill out an entire section and in 
others they selectively answered a subset of questions. Surveys with only a few missing 
responses were retained in the analysis. In the analysis of individual items, data were 
omitted casewise. In the formation of scales missing data were imputed using item mean 
values. 

From the 58 items five scales were formed representing; 1) general student instructional 
opportunities related to the 1992 Framework and the CBAAM (14 items); 2) content exposure 
linked to the 1994 CBAAM (6 items); 3) teacher actions and opinions that illustrate alignment 
with the 1992 Frameworks and the CBAAM (12 items); 4) a scale inquiring about 
opportunities, desires, and confidence with respect to one's current professional development 
in mathematics (9 items); and 5) classroom assessment practices (10 items). Seven items were 
not included in the scale analys* because they lacked clear content alignment with the five 
scales. 

Two groups of teachers were identified from the surveys; those with no involvement in the 
CBAAM development or scoring, and those that participated in the 1993 or 1994 
development or scoring activities. Independent t-tests were calculated to compare mean 
differences between the groups on each of the survey items and the five subscales, 
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Interview Procedure 

Interview Participants. Twelve teachers were selected to participate in individual interviews. 
One dropped out in the final stages of the data collection and was not replaced. The primary 
criterion for selection was level of involvement in the CBAAM development process. Four 
selected teachers had no involvement with the CBAAM development, or scoring. A second 
group of four had experience scoring student work, but was not involved in any of the 
assessment development. The final group of three was involved in the CBAAM item 
development, piloting, and scoring. This stratification was important in order to obtain a 
cross-section of data from teachers with different levels of investment in the CBAAM. 
Interview participants represented seven schools. All teachers participated from one building 
(n=4). This provided a better opportunity to evaluate within arid across grade-level impact at 
this site. 

Interview Procedure. Individual interviews were scheduled in the late afternoons and 
conducted in classrooms at each participating teachers site. Interviews took between 60 and 
120 minutes and teachers were paid a $40.00 stipend for their participation. The interviews 
were conducted between March 24 and April 20, 1994. The interviews were designed as an 
open discussion about teaching and assessment practices, the CBAAM, professional 
development and interaction with peers in mathematics, and state reforms including the 
frameworks and related assessments. The guiding questions presented in Appendix B served 
as a starting point for the discussions, but did not necessarily determine the topics or 
sequence of topics during the interviews. All interviews were audiotaped and transcribed. 
Third and sixth grade development committees were invited to a debriefing on June 2, 1994 
where results of this research was presented and discussed with them. 

Interview Analysis. The process of analyzing the teacher interview data involved the 
following steps: (1) transcription of the tapes, (2) identification of themes, (3) content 
analysis of the transcripts, and (4) cross-teacher analysis of comments within themes. 

Transcription of Tapes. After the interviews were taped, verbatim transcriptions were 
completed for each teachers interview. Transcribers signified unintelligible language by 
putting ambiguous words or phrases in parentheses. If the transcriber was entirely unable to 
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understand the teacher s or interviewer's words, then ellipses were inserted for unintelligible 
words and phrases. In these verbatim transcriptions a unique identification code was 
assigned to each teacher. Each teacher's comments were transcribed in their entirety. 

Identification of Themes* Once the interviews were transcribed, they were read in 
their entirety by two researchers familiar with the study and the CBAAM. Upon reading 
each interv iew, the researchers replaced unintelligible words and phrases if they were certain 
of their interpretation. The researchers also corrected the spelling of misspelled words. For 
the final presentation in this report repetitions (e.g., I think.. .1 think... I think) and 
unintelligible speech (e.g., ummm...y...y...aaaa...waa) were, eliminated to enhance the 
readability of the discourse presented. Otherwise, the transcripts were not altered. 

After all the transcripts had been read, the researchers identified common themes that 
teachers addressed during the interview process. For example, many teachers described the 
way they reordered their curriculum to prepare students for the CBAAM. Thus, 
"resequencing of content" was identified as an important theme. Another commonly voiced 
topic was teachers use of rubrics to articulate performance standards to their students. Thus, 
"use of rubrics" was also an identified theme. Each researcher identified the common themes 
independently and then discussed their lists. Based upon this discussion, the most central 
themes were identified. 

Content Analysis of the Transcripts. Once the list of themes were identified, 
interview transcripts were coded. For each transcript, each instance of a theme was 
identified and designated with a unique code. During the analysis, the following themes 
were identified: 

♦ the impact of the CBAAM on the content, sequence,, and process of classroom 
instruction; 

♦ teacher use of the CBAAM results; 

♦ use of the CBAAM problems, rubrics, and results as a device to inform parents about 
the new mathematics program; 

♦ use of rubrics as a device to articulate performance standards to students, teachers 
and parents; 

15 



21 



♦ use/nonuse of standardized tests; 

♦ professional development opportunities in mathematics; 

♦ the California Mathematics Framework; 

♦ articulation of the CBAAM: within grades; across grades; with the district curriculum; 
with the California Frameworks; 

♦ fairness of the CBAAM, in particular non-standardized administration procedures, 
amount of classroom preparation, and teaching the specific content that will be 
assessed; 

♦ what the CBAAM does and does not measure; 

♦ problems with the CBAAM, including its developmental inappropriateness, heavy 
writing burden, and unclear directions. 

Analysis of Teacher Comments. Teacher quotes were identified representing each of 
the above themes. Many of the themes were directly linked to the research questions, such 
as explanations of instructional modifications anu discussions of instructional guidance. 
Much of the discourse on these topics was stimulated by the interview procedure. Other 
themes were indirectly related to the research questions and primarily teacher generated. 
Examples of these include, comparisons of the CBAAM and the MAT-6, and teacher concerns 
about procedural issues associated with the CBAAM. To facilitate the cross-case analysis, a 
catalog of quotes generated by all 11 teaches was compiled for each of the themes. The 
catalog was used to identify teacher perspectives on the themes listed. Many times teachers 
differed on a given issue; in these cases both sides of the issues are presented. The catalog of 
quotes was a comprehensive source recording each teacher s perspective on a given theme. 
Conclusions based on the interviews are provided in the Results section. Supporting quotes 
from the interview and survey data provide evidence for the conclusions drawn. This 
research focused on cross-case analyses of the themes. In no case do statements by a single 
teacher sei*ve as evidence of a theme. 
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IV. Survey Results and Discussion 



Individual Item Results 

Table 1 in this section presents the survey results for individual items. The first mean 
represents the performance of 20 teachers who were not involved in the CBAAM 
development or scoring. The second mean represents the performance of 29 teachers who 
were involved in the CBAAM development or scoring. The content of each item is presented 
with the corresponding group means and significance level if equal to or less than .05. There 
were significant group differences on 17 of the 58 individual items. For all 17 items the 
CBAAM involved group had higher mean values. This could be interpreted as a function of 
a systematic positive response bias for those teachers involved in the CBAAM project. 
However, on several survey items that were included to correspond to "traditional" 
instructional approaches, teachers who were involved less with the CBAAM rated higher. 
Examples of this include Items A6, A14, and B7. Significant differences on any of these items 
in favor of the CBAAM involved teachers would elevate concerns of a systematic positive 
response bias. Additionally, the majority of the CBAAM involved group voluntarily scored 
the assessments, but had not invested in the actual item development or field testing. 
Therefore, it is unlikely that their ratings indicate support for their own development efforts. 

The CBAAM involved teachers reported more frequent instructional use of open-ended 
problems, investigations, cooperativ * working groups, and the use of literal and figurative 
tools to aid conceptual understanding. They placed greater value in the CBAAM initiative 
and the results obtained from the assessment. They also reported closer alignment between 
the CBAAM and their own instructional strategies, and indicated a greater desire for peer 
interaction and professional development in mathematics. Involved teachers also reported 
more frequent and varied use of alternative assessments. Significant differences between the 
item means in Section C of the survey were consistent with significant differences in 
Section A. Open-ended tasks, investigations, portfolios, projects, rubrics, and student self- 
assessment were used more frequently by teachers involved in the CBAAM. In general, 
involved teachers reported more movement in classroom activities and assessment practices 
consistent with changes suggested by the instructional guidance presented in Frameworks, 
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NCTM Standards, CLAS and the CPA AM. One interesting result occurred on items All and 
CIO; "students use scoring rubrics or scoring criteria to evaluate their own work." Although, 
involved teachers reported significantly more use, both groups exhibited minimal use of 
these tools as instructional devices. It is not surprising to find infrequent rubric use in 
classrooms by teachers who report lower levels of open-ended /investigative task use and 
who are not involved in scoring student work on the district assessment. However, greater 
than monthly use of "rubrics" would be expected of teachers reporting regular classre-om use 
of complex-rormat tasks, especially after participating in district activities that employ the use 
of rubrics to score work. Group differences on individual items revealed some interesting 
results, but the primary purpose of the survey was to evaluate group differences on clusters 
of related items, or scales. 

Scale Results 

The survey was composed of five subscales which were used to compare the two groups of 
teachers. Table 2 presents the five scales, internal consistency reliability, means, standard 
deviations, and significance levels from two-group comparisons. Based on self-report, 
teachers involved in the CBAAM engaged students more frequently in a variety of 
framework consistent activities, have stronger alignment in philosophy and teaching 
behaviors with the framework and new assessments, have greater interest and desire for 
professional development, and use alternative assessments more frequently. The most 
interesting two-group scale comparison arises from the fact that groups do not differ on the 
Content Exposure scale. When asked about frequency of teaching content directly related to 
three of the CBAAM items, both involved and non-involved teachers provided similar 
student opportunities. Students were generally provided "more than two lessons" on the 
content related to each item. Announcement of assessment content and the related teacher 
meetings apparently had the intended effect; regardless of their predisposition towards the 
CBAAM, or the type of instructional processes teachers reported, or their involvement in the 
CBAAM development and scoring, teachers specifically prepared students for the assessment. 

This outcome is not unique, considering the body of research available that suggests teachers 
regularly engage in test preparation with their students (Moore, 1994). They do this for a 
number of reasons, including, insuring student exposure to content, test procedures, and test 
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format. They are also motivated by stakes connected to student performance, district 
demands to increase scores, personal demands to perform well in comparison to other 
teachers, and a genuine concern- for their students. Elementary children are much more 
dependent on their teachers for knowledge exposure and guidance related to the actual 
assessment situation. This may stimulate teachers to provide much more direct support s > 
that students are adequately prepared for the assessment situation. 

Moore (1994) also found higher levels of content preparation relative to the process 
preparation. In Mt. Diablo, where teachers only had six weeks advance notice, it would be 
reasonable to interject a few specifically designed lessons or tasks focusing on the content of 
coordinate systems for example. It is another matter to fundamentally alter the patterns of 
instruction that evolve over many years. What the survey does establish is that there is a 
correlation between the CBAAM involvement and changing patterns of instruction. The data 
do not identify the CBAAM as the cause of that change. Two scenarios are likely; 1) teachers 
already making instructional /assessment changes volunteer to participate in the CBAAM 
activities, or 2) the CBAAM participation motivates changes in classroom 
assessment/instruction. In either case, the CBAAM involvement can be viewed as a process 
that facilitates that change process; either by recognizing, reinforcing and aligning with 
instructional changes in classrooms, or leveraging the change process by illustrating and 
communicating the content and process of desired modifications in mathematics instruction. 

Open-Ended Item Results 

Teacher comments on the open-ended survey question supported the premise that, regardless 
of teacher disposition toward the CBAAM, they prepared their students for the content on 
the assessment. The responses were quite diverse and either supported or refuted the 
validity and utility of the assessment in terms of the direct implications for instruction or the 
consequences of the information provided by the CBAAM. Consider the following extreme 
examples: 

The CBAAM has been influential in my teaching - by the district 
instituting such a test it has made me aware that the district is serious 
about teaching problem solving and discrete mathematics in the classroom. 
It has been a wonderful tool for me to use to assess students' progress in 
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this area, and most importantly, my progress in teaching students. As a result of 
the CBAAM, parents are now starting to understand why there are less dittos and 
worksheets coming home; they are also beginning to see how they can better 
support their children. 

The CBAAM was a horrifying experience in my classroom. We are 
required to give the test in January, a time when students and teachers are 
just beginning to get their feet wet. I do not teach fractions, geometry, or 
telling time until later in the year. With thi CBAAM, however, I must 
change the timing of my curriculum presentation. Unfortunately, most 
third graders here at my school are not ready to take on fractions, 
geometry, etc.. .until later in the year. I spent several weeks preparing my 
studerUs for the CBAAM. I feel they were robbed of the quality teaching 
time they could have received if the test were given later in the year. 

The polarity and diversity of the survey comments helped guide the interview discussions to 
focus on what specifically happens with teachers and students as a result of the CBAAM. 
Why does the same assessment, in the same schools, with similar groups of children; result 
in such radically different perceived outcomes? What mediates this? In the interviews, 
discussions centered on descriptions of teacher and student actions and behaviors directly 
linked to the CBAAM, use of the CBAAM information, use of the Frameworks, professional 
development experiences, and specific examples of classroom instruction. This helped to 
remove some of the emotional or philosophical flavor of comments that made it difficult to 
interpret the impact of the CBAAM on teaching and instruction for students. 
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TABLE 1 

Means, and two tailed significance levels fcr teachers that were 
involved, versus not involved in the CBAAM development or scoring. 
A. Student Experiences 



Please use the scale below to estimate the frequency 
of student participation in the following activities 
in your mathematics program. 

Nfit CBAAM 
Involved Involved 



Item X (n»20) X (n«29) Sig. 



1, Students write about their mathematical thinking 
to support and clarify their solutions. 


3.00 


. 3,48 


ns 


2. Students work together on problems, sharing and 
discussing ideas, strategies, and solutions. 


3.50 


4.14 


p<.05 


3. Students use tools such as manipulatives, 
calculators, computers, and rulers. 


3,95 


4.41 


p<.05 


4. Problems and instructional activities require 
students to apply mathematical ideas to the world 
beyond the classroom. 


3.50 


3.93 


ns 


5. Students use visual representations, 
computational algorithms, and models to 
to illustrate math thinking. 


2.55 


3,76 


p<.01 


6. Students work independently to complete 
problems focusing on mastery of specific 
computational skills. 


4.10 


. 3.90 


ns 


7. Students are exposed to complex open-ended 
tasks that have multiple solutions. 


3.05 


3.97 


p<.01 


8. Students complete tasks that involve investigating 
mathematical ideas over extended periods of time. 
Problems and related activities may involve several 
class periods over several days. 


2.20 


2.65 


ns 


9. Students reflect on their work (orally or in writing) 
to express new learning or understandings. 


3.40 


3.58 


ns 


10, Homework involves formulating, completing, or 
revising open-ended tasks. 


2.40 


2.52 


ns 
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TABLE 1 

Means, and two tailed significance levels for teachers that were 
involved, versus not involved in the CBAAM development or scoring. 
A. Student Experiences Continued 



Please use the scale below to estimate the frequency 
of student participation in the following activities 
in your mathematics program. 

Not CBAAM 
Involved Involved 



Item X (n»20) X (n=29) Sig. 



11. Students use scoring rubrics or scoring criteria to 
evaluate their own work. 


1.15 


1.93 


p<.01 


12. Students revisit and revise their 

solutions/responses to open-ended problems. 


1.70 


2.17 


ns 


13. Mathematics is integrated into instructional 
activities in other subject areas. 


3.30 


3.62 


ns 


14. Homework involves problem sets that reinforce 
basic skills. 


3.65 


3.59 


ns 


15. Students represent and compare quantities using 
manipulatives and visual representations. 


4.30 


4.45 


ns 


16. Students collect, organize, and represent data 
in charts, tables or graphs. They write about 
their analysis and interpretation of data. 


4.05 


4.31 


ns 


17. Classroom activities deal with understanding and 
using coordinates and coordinate systems. 


4.21 


4.31 


ns 


18. Classroom activities focus on part-to-whole 
relationships and visual representations of 
fractions. 


3.90 


4.00 


ns 


19. Students create and read maps (for example, they 
use directions, locate landmarks, estimate distance, 
and interpret legends.) 


4.40 


4.52 


ns 


20. Students create and interpret schedules involving 
time and other dimensions. 


3.30 


3.76 


ns 
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TABLE 1 

Means, and two tailed significance levels for teachers that were 
involved, versus not involved in the CBAAM development or scoring. 
B. Teacher Experiences/ Opinion 



Please read the following statements' and circle 
the response that best indicates your view. 

Not CBAAM 

Involved Involved 



Item X (n=20) X (n=29) Sig. 



1 It is important to meet with a group or team of 
teachers regularly to discuss mathematics content and 
instructional strategies. 


4.10 


4.41 


ns 


2. I have altered the way students 'do " mathematics to 
correspond to the 1992 Mathematics Framework. (For 
example, collaborative work, writing about math, and 

h* tended tacWc or investigation O 


4.05 


4.34 


ns 


3. 1 have ample opportunity to review and discuss examples of 
problems and student work related to the content and 

cfanHarHQ in the 1 QQ9 K^atHpmaHrc Framework 
sicuiuaius ui uic 1774 iriauicuiauu i lauisvvuiiK. 


2.65 


2.86 


ns 


4. In the past two years, the district has provided 

nrofpQQional Hevplonmenf onnortiiniHpQ that 

enhanced my understanding and teaching of 
mathematics. 


2.95 


3.10 


ns 


5. 1 have altered the content of my instructional 
units to correspond to content covered by the 
CBAAM. 


3.80 


4.10 


ns 


6. There is adequate time to teach basic computational skills 
and more complex problem solving. 


2.75 


2.79 


ns 


7. Textbooks are the primary source of instructional 
guidance and classroom activities. 


2.40 


2.00 


ns 


8. In my class, I have altered the way students 
"do" mathematics to correspond to the tasks on 
the CBAAM. 


3.15 


3.52 


ns 


9. Classroom activities are often based on the application of 
mathematics to real world situations. 


380 


3.79 


ns 


10. 1 am confident in my ability to develop and 
use complex problems and investigations in 
my teaching. 


3.65 


3.62 


ns 
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TABLE 1 

Means, and two tailed significance levels for teachers that were 
involved, versus not involved in the CBAAM development or scoring. 
B. Teacher Experiences/ Opinion Cont. 



Please read the following statements and circle 

the response that best indicates your view. Not CBAAM 

IflYQlved Involved 



Item X (n«20) X (n«29) Sig. 



11. Teachers were clearly informed of the general 
content and format of tasks on the CBAAM. 


3.90 


3.93 


ns 


12. 1 think recent changes in mathematics instruction and 
a^se^sment are necessary and important. 


3.60 


4.03 


ns 


13. 1 have altered the content of my instructional units 
to correspond to content described in the 1992 
Mathematics Framework. 


3.60 


4.00 


ns 


14. Feedback about student performance on the CBAAM 
will be valuable in my future instructional planning. 


3.30 


3.93 


p<.05 I! 


15. Prior to CBAAM administration, I reviewed the 
generic scoring rubric with my students. 


2.60 


3.03 


ns 


16. Tasks on the CBAAM mesh well with the type of 
problems students are exposed to in my math 
classes. 


2.65 


3.31 


p<.05 


17. In recent years, I have been involved in valuable 
professional development experiences (in math) 
outside of the district. 


2.45 


3.27 


p<.05 


18. Students will benefit by reviewing and 

understanding their performance on the CBAAM. 


3.25 


3.82 


ns 


19. My students were well prepared for the content 
and task format on the CBAAM. 


3.05 


3.14 


ns 


20. The time and effort spent developing and scoring 
the CBAAM is worthwhile in terms of the benefits to 
teachers and students. 


2.70 


3.44 


p<.05 


21. The district should only use assessment results to 
provide feedback to individual students and teachers. 


3.65 


3.55 


ns | 


22. Administration of the L3AAM and similar statewide 
assessments will motivate teachers to change 
instructional practices. 


3.15 


3.69 


ns 


23. 1 would like to have more systematic interaction 
with other teachers that are developing and using 
alternative assessments. 


3.60 


4.17 


ns 
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TABLE 1 

Means, and two tailed significance levels for teachers that were 

involved, versus not involved in the CBAAM development or scoring. 

C Classroom Assessment 

Please use the scale to indicate how often 
you use the assessment tools listed below. 

Nflt CBAAM 
Involved Involved 

Item X (n«20) X (n«29) Sig. 



1. Open-ended tasks 


3.39 


4.07 


p<.01 


2. Portfolio 


2.85 


3.48 


p<.05 


3. Textbook tests and quizzes 


2.95 


3.27 


ns 


4. Teacher-made tests and quizzes 


2.50 


2.58 


ns 


5. Developmental scales 


2.55 


2.62 


ns 


6. Investigations 


3.05 


3.55 


p<.05 


7. Projects 


2.55 


3.31 


p<.01 


8. Exhibitions 


2.20 


2.62 


ns 


9. Anecdotal records 


2.45 


3.31 


p<.01 


10. Holistic or analytic scoring rubrics 


1.95 


2.65 


p<.05 


11. Observational checklists 


2.95 


3.20 


ns 


12. Student self-assessment or reflection 


2.65 


3.31 


p<.05 


13. Oral questions/interviews 


3.40 


3.37 


ns 


14. Attitude surveys 


1.95 


2.20 


ns 


15. Peer assessment 


2.10 


2.83 


p<.05 
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TABLE 2 

Group means, standards deviations, 
internal consistency r eliability (RXX 1 ) 
and two-tailed significance for scales constructed from survey items. 



Not Involved CBAAM Involved 



SCALE Rxx 1 X SD X SD Sig. 



Student ExDeriences 
Reflecting Framework 
(14 items) 


.80 


38.05 


6.64 


44.69 


7.80 


P<.01 


Content Exposure 
Related To CBAAM Questions 
(6 items) 


.68 


24.15 


2.89 


25.30 


3.17 


ns 


Teacher Alignment 
With Framework /CBAAM 
(12 items) 


.81 


40.45 


5.63 


45.03 


6.55 


P=.01 


Professional Development 
Interest/Opportunities 
(9 items) 


.75 


28.90 


5.49 


32.62 


4.18 


P=.01 


Assessment Activities 
(10 items) 


.84 


25.70 


4.26 


31.66 


6.58 


P<.01 
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V. Interview Results and Discussion 



In this section, evidence related to the themes of the interview discourse is presented in clusters of 
teacher comments. This was done to illustrate to the reader the nature of the evidence bearing on 
the essence and magnitude of the effects characterized by each theme. Some effects are strong and 
supported by many diverse responses, while others are important but not distributed across 
teachers. Occasionally, interviewer comments are included to provide context. In some cases, 
survey comments are included to buttress the arguments presented. 

Teacher quotes are labeled throughout this section to indicate group assignment and individual 
teachers. For Example, El refers to teacher number one who had experience with the CBAAM 
development or scoring, while Nil refers to teacher number 11 who did not have experience with 
the CBAAM development or scoring process. Because different quotes from the same teachers are 
used throughout the discussion, labels El, Nil, etc., identify all quotes arising from an individual 
teacher's protocol. A participant reference number above 11 (e.g., E14, N40) indicates the quote is 
from a survey comment. 

The CBAAM Impact on Third Grade Curriculum and Instruction 

The interview data confirmed the survey results and indicated that teachers altered their instruction 
in specific ways in order to introduce content covered on the CBAAM. Because the CBAAM was 
moved from April to January, many teachers responded by resequencing existing units and content 
strands to provide student exposure prior to the mid-year assessment. Other teachers introduced 
new content that was not planned or included in theu third grade program in prior years in order 
to accommodate the CBAAM content. All teachers reported some effort to introduce specific 
CBAAM content regardless of their level of participation in the CBAAM project. However, 
comments from teachers with experience in item development or scoring described smoother 
transitions when introducing new content and projected a stronger philosophical alignment with the 
changes. Alternately, the content transitions were more difficult for the less involved teachers and 
they often expressed frustration with the preparation and assessment process. 

Instructional Sequence. Resequencing the third grade curriculum to prepare students for the 
assessment occurred in many ways. For example, typically teachers would have introduced . 
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fractions, charting, and coordinates in the spring. For most teachers with or without the 
CBAAM experience, content that was normally introduced late in the second semester was 
interjected without apparent deleterious effect earlier in the academic year: 
El 

Sa..not that I was teaching to the test.., but I realized that the things the test 
would be locking for in January was what I didn't usually get to until April. .and 
so I had to redo my schedule. And also, my two colleagues said we need to move 
some of this stuff up front and do it early. Whereas, usually you do your basics 
the first part of the year. That (the CBAAM content) comes at the end of the year 
zvhen you feel like they've.. .really done a good job.. .and actually though, by 
turning it around Ms year I don't feel that they are any worse off. In fact they 
may be better off. Because we did all of those things... the fun stuff,. .starting in 
September. And I kind of eased in the hard. .dull things that would normally be 
first. 

E3 

In some cases I thought they would do a little bit better than they had done. 
Because I thought that by doing what I'm doing in the classroom md by 
preparing them with knowledge of the context that were going to be involved 
with in the CBAAM, Ms would give them a fair shot And what it just 
showed me is that starting in September is not soon enough, so I was 
disappointed in how they had done, but it taught me. 

E4 

/ think that it does reflect what we do in the third grade. I think it definitely does. 
Well it depends on...what time of the year you give the test. The fraction problem 
I am just going to get to fractions next, well within two xveek.J just sort of did 
some quickie lessons ahead of time to get them prepared for it. But I think if the 
test were given at the end of the year then all the kids would have done better on 
it. But that's just because I wouldn't get to fractions until later on in the year. I 
spend more time on multiplication and adding and subtracting. 

"J. 5 

£5: Well for example on the grid question we hadn't done anything with grids so 
we just sort of pulled together third grade teachers and pulled out all the materials 
that we had (bout it, and then we did several different lessons on grids and how to 
read it and., say what the coordinates were and that kind of thing. 

I: You wouldn't of done that otherwise if it wasn't on the CBAAM? 

E5: No, that's right. 

I: Would you of done it in the third grade at all? 

E5: Yeak There is some material we would of done. Although not necessarily at 
that time. 
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E6 

We were right in the middle of the Marilyn Burns multiplication unit when 
la teacher] cane hack and said, * the CBAAM it's on coordinates. 00 Oh my God we 
hadn 't done any coordinates and we were really good at that (in past years). So I 
dropped everything and went into coordinates. 

E7 

/ mean. ..we did do coordinates. The only,. .problem I ran into was that graphing is 
something I'm going to be doing but...I was sort of planning on doing it near the 
end of the year.. And the CBAAM came sooner, so I wanted to give them some 
kind of an experience with it. So toe did them earlier. 

Many of the teachers who had not been on development teams or scored student work in 
past years, presented varying levels of frustration related to resequencing their instruction. 
Their information about the assessment was primarily derived from the CBAAM teacher 
in-service in December, 1993. Had they known about the content at the beginning of the 
year (or earlier), the transitions might have been less cumbersome. As it was, they expressed 
a sense of being taken by surprise, but they were still compelled to make changes to provide 
students with pre-assessment opportunities. 
N9 

Well because I think I had more time (in past years)...not necessarily to prepare 
children for the test. But more time to cover Hie material that was on the test. 
Fractions are okay.. .if you stick to halves, because children can understand that 
concept. But fractions for third graders don 0 t come until springtime. Then you 
give it to them in January? You have to take everything out of sequence to get it 
in, to cover that material. I mean you had to push everything else back.. Well a 
good third of the year is efter the test And fractions don t come until that part of 
the year. 

N10 

All of a sudden we had to step up what we were doing because we were not going 
to get into geometry that quickly. 

I: You took some things you were doing and you just moved them up? 
N10: That's exactly it. 
I: Did that create... 

N10: That didn't create any problems really, but 1 fust thought well 

usually. ..getting into geometry and fractions comes later in the year and not in 

January. 
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N16 

Although I consider my strategies current xvith the new framework, I did 
have to spend lots of time teaching ahead of where the students currently 
were developmentally, especially in the scheduling and coordinate plotting. 

N41 

We were told of the CBAAM (content) just before winter break We started 
giving, the test in January. We should hope had much more time to prepare 
ourselves and our students. We need, during school time, to work on 
methods. 

N46 

In general, I like the move toward more cooperat iv e learning and writing in 
mathematics. I question the objectivity in scoring these types of "tests" and 
the level of cbility the "tests*' require. I really had to drop everything and 
teach to the test. 

Instructional Content. In addition to resequencing existing content teachers also described 
how they introduced new content that was not typically included in their third grade 
mathematics program. Generally, this was not consistent across teachers, but reflected a 
"filling in" of content individual teachers had not traditionally introduced. However, most 
teachers expressed that the carnival problem, which dealt with creating a time by activity 
schedule, was completely new content. A number of teachers also expressed reservations 
about introducing content that was too cognitively advanced for third graders or not a part 
of the explicit curriculum of the district. 
N8 

/; When you received information (bout the items what exactly did you do? 

N8: We went through examples of graphing...cf mapping. They gave us 
some worksheets we talked (tout it, we played.. .those kind of games so they 
were familiar with the format And they actually did better on that section of 
the test than I would have expected them ta I just, I felt that it. .wasn't 
sometldng that the third grade emphasizes, so Hie teaching that I was doing 
was taking time out from the things that I should of been covering. 

N9 

I: Do you think your introduction to schedules prior to the assessment 
were. ..some of their first experiences? 

N9: Definitely.. .and we took schedules from the newspaper, and we took 
schedules from the television and movies, and things that they might be 
familiar xvith and.. .baseball schedules. But that was too broad a lock It was 
a look at too many different kinds Of schedules which tended to confuse them. 

h If it's their first time...their first real opportunity. 



N9: It was their first exposure. 1 think scheduling is beyond...the scope of 
ivliat a third grader can do in January. It may be beyond the scope of what 
ttiey can do in May. Because it just isn't in their frame of reference at ail 

E5 

£5; Now the carnival problem, that was a red problem because I don't 
think the average third grader can even read a schedule necessarily prepare 
one. I don't think they're even interested in schedules. Certainly, they 
wouldn't be interested in making up a schedule that they weren't required 
to. So what we did is...we took some schedules out of newspapers, well 
mostly the newspaper. We also found the TV Guide schedule., .how do you 
tell when a program is on and what channel and that kind of thing? 

h Did they actually have to create one? 

E5: Yes, I went through and created one with them. Solve did it more or less as a 
class assignment because there wasn't time for that kind of thing. We brainstormed 
a field trip, and what would you need for the group. 

Teachers who did not alter content or instructional sequence were the exception. Several 
teacher comments from the surveys and interviews discussed "misalignment between third 
grade curriculum and the CBAAM items, but most described content changes in spite of this. 
Several teachers expressed frustration about significantly altering content to correspond to 
the CBAAM. Their statements reveal concerns about "teaching to the test" The description 
below characterizes one teacher's dilemma between following explicit district curriculum 
guidelines that do not address the CBAAM content and introducing the new content at the 
risk of not covering what appears in the guidelines. 
Nil 

The assessment is very poor at my grade level It doesn't test what I'm told 
to teach. My math program is based on the third grade strands of 
achievement The district tells me that my job is to teach, in math, these 
things (referring to a district mathematics curriculum guide). 

I don 't chmge my content to match the CBAAM because I feel strongly 
(gainst doing that when its not the right content I felt really strongly that 
with the CBAAM. ..to take time ta..either cram or do a crash course when it * 
wasn't a third grade expectation. 

It's really frustrating to me when I try hard to cover the concepts at third 
grade level and then the test covers ether topics. I'm left with two choices; 
1) allow my students to honestly take the test and be unsuccessful and feel 
poorly because they haven 't been taught, or 2) spend a week teaching to the 
test so that my students will be Me to do it This allows them to feel 
successful, but it makes the results meaningless and invalid. This year I 
chose number two. 
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Developmental Level of the CBAAM Items. A more pervasive teacher concern related to 
CBAAM content involved the perceived developmental level of the items. Comments about 
specific items from both interview and survey protocols revealed that teachers, regardless of 
their experience or opinion of the CBAAM, questioned the cognitive complexity of items. 
This was especially true for the "Carnival" item requiring students to develop a schedule, and 
less so for the Treasure Hunt" item involving coordinates and plotting. 
El 

I was not too happy with ihe "Carnival " item. I felt it was fourth grade 
level Well they cm't even read a T.V. guide, a TV. schedule. Its very 
hard too for them to sit down and make a schedule. Now they live by 
schedules all the time. But they don't redly know it in that frame, that 
term. I went back and I said okay, what time do you get up? What time 
do you eat breakfast? We went that route. But it was very hard to make 
the connection. And I find that the ones who did it had to have some very 
distinct guidelines to go by. I don't think they were developmentally 
ready. 

E3 

It involved so mmy different modes of thinking. They have the 
understanding of a schedule and I had thought I had given them a 
background But then actually...a tdble and then having to think of three 
different factors at the same time. This would be something diffiadtfor 
my husband to do. 

E4 

We gave it right after we got back [from winter break]. Sow had a 
couple of weeks. And we did a lot of graphing activities and scheduling. 
But even with the scheduling activities...! did an activity where the kids 
had to do almost the exact same problem where I told them that we were 
going to have a party in the classroom, that we were to raise money, a 
similar kind of problem, and that the jobs would be greeting people at the 
door, pissing out cooties md serving refreshments. And that they had to 
decide who was going to show up md all I thought that was too little 
structure for the kids and it was really hard for them to grasp the concept. 
And most cf them I'd say were Ale to do it tfler a couple of tries. I still 
don't really think that they understood. And when I was scoring he tests 
I could see that kids were just solving the problems the way that their 
teachers htd taught them to. Just the same with my class. 

E5 

The scheduling was a lot harder. Because it's something they never...deal 
with. Well, your average third grader cctnot tell time for one thing. So 
that's a problem. 
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E7 

I think in general the teachers like the idea of this kind of a test, although 
I've Iteard complaints about certain problems that were on the CBAAM 
tins year. Like the schedule for example. A lot of teachers just felt that it 
was too difficult. We do a lot of things with multiplying and subtracting, 
and we do a lot of writing on utiat subtraction is, and I guess we felt like 
tlie schedule was something that was just so open-ended it was tough for 
the kids. * ' 

E20 

/ was very pleased with the test this year. However my students had 
difficulty creating a schedule because in their lives they have never had to 
make a schedule, only read them. 

Based on the analysis of teacher comments, criticism from the CBAAM involved teachers 
tended to be less intense than those not involved in development and scoring activities. 
However, affect aside, most teachers questioned the complexity of the "Carnival" item in 
p?rticular, but still altered their instruction in order to prepare students for the CBAAM 
content. 

N8 

Tint's my real criticism. It's not the philosophy behind it, it's the choice of 
items. I think that the CBAAM didn't measure what the kids could show 
in...graphing. And graplting is something that we cover. The scheduling item I 
think is outside of the third grade curriculum. It dealt with time, it dealt with 
time over the twelve o'clock hour. Which is a...fairly sophisticated concept for 
some of tlte children. It wasn 't just measuring just a four hour time span, you 
had to go from ten to twelve, and twelve to two, and realize you know what that 
ti>.te was. I'm not sure that these children would be expected to make up a 
schedule for a booth for four periods. I don't think that was a real world 
application for them. I think there were too many variables for them to deal with. 
The "Hershey Bar " one I think was a good example of something that fits into the 
curriculum. Something the children didn't have a problem with. Something that 
they could handle and explain very well The other two I found I had to do a lot 
of pre-Uaching to get them ready to even take test. 

N9 

N9: The weakest area for my class was the "Carnival." And we had talked 
about schedules and things like that. 

I: Why do you think it was so hard for them? 

N9: 1 think that particular concept is beyond the third grade. I don't 
think kids have...practical exposure...with a T.V. guide even. Mom and 
Dad read it, and then the kids get it as far as the schedule is concerned. A 
movie schedule,...a bus schedule. Third graders don 't read it themselves. 
They're not exposed ta..the making of the schedule. 
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N10 

I think overwhelming...fir$t of all they had to figure out a schedule and 
that's very difficult for third grade lads, because future time is like from 
new until lunchtime, and to logically figure out hew they're going to 
staff the different jobs at the carnival and to justify it? You knew they 
were keeping so many different variables in their head, it was very difficult 
for them. Development ally very inappropriate, especially the item that 
dealt witk..the "Carnival" and the scheduling. 

N12 

The items on this year's CBAAM were develcpmentally, totally 
inappropriate. According to research, the items, especially the "Carnxvd" 
scheduling were seventh grade. The' scoring rubric will be meaningless 
because the ihms were so far off what eight year olds can cqgnitively da 

Many teachers blended the idea of developmental inappropriateness with the assessment 
being inconsistent with district curriculum guidelines and the historical sequence of content 
coverage in the third grade. This could be interpreted as resistance to change or a lack of 
philosophical alignment with the assessment. However, the teachers were confronted with 
many simultaneous changes. New content had to be introduced in a new sequence, and for 
many they had little time to prepare for these changes. In addition, many third grade 
students had truly not mastered concepts or skills tapped by the assessment. 
E5 

The test items were developmentally inappropriate. Coordinate geometry is 
not part of the third grade curriculum. Fractions are only part cf the end 
of the third grade curriculum, and we gave the test in January. 
Scheduling is totally inappropriate, one half cf the third grade children 
cannot tell time - let alone figure out how to cross the 12:00 hour. They 
are not interested <md do not deal with them (schedules). 

N36 

The assessment uses practical knowledge md organizational skills. They 
expect every child has had practical experience in making a schedule or 
planning a route or dividing fractional numbers. They need these 
experiences, but seldom are ready for the CBAAM assessments unless 
preteadmtg is done. Third grade is an eight or nine year old, not m 11 or 
12 yem old child. Given the same test to fifth grade students, our results 
would not be from a teacher taught assessment, as I feel we are doing with 
third graders. 

The above examples point to problems related to student development and mastery of 
concepts (i.e., telling time) related to the CBAAM items. They also illustrate an assumption 
that developmental readiness is correlated with curriculum sequence. In this situation, if all 
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the children were developmental^ ready for the items, they still had a limited window of 
opportunity to experience the content. Teachers took advantage of that window, but often 
found it was too limited an opportunity for their students. Consider the "Treasure Hunt" 
(see Appendix A) as one example. In reality, children can begin learning about spatial 
relationships through organized two dimensional grids (geoboards) as early as kindergarten. 
In subsequent grades, they can develop locating and mapping skills by learning direction, 
distance, and by visualizing location using coordinate grids. It is these multiple 
opportunities and experiences over time that build a child's mastery of these concepts; not 
the introduction of a single unit in the third grade. The assumption that these skills are 
taught and learned in only the third grade would wreak havoc with most third grade 
mathematics programs. The introduction of special units and subsequently the CBAAM 
assessment should be viewed as the next step in the teaching /learning process and not the 
final evaluation based on one limited exposure. 

All teachers interviewed reported their efforts to cover the content being assessed by the 
CBAAM items prior to the assessment. As part of their preparation they reordered content, 
and added new content. Teachers also stressed instructional processes that support the 
California Mathematics Framework and the NCTM Standards. By working hard to prepare 
students for the test they increased students' opportunities to learn the mathematics content, 
and to participate in extended investigations that represent complex problem-solving 
situations. As they prepared for the CBAAM, students had more opportunities to work in 
cooperative groups, use manipulatives and engage in hands-on math activities. Students 
were also asked by teachers to be more exphcit about how they solved mathematics 
problems. 



I nstruct i on a l PrormM , Although teachers reported substantial changes in their instructional 
sequence and content, most teachers, in preparing for the CBAAM, did not have to step 
outside of established teaching methods and strategies. When preparing for the CBAAM, 
they did not significantly alter the process of instruction. They used; open-ended questions 
and investigations, cooperative learning groups, manipulatives and extended oral and written 
responses to questions. Teachers reported that most of these strategies had been in their 
instructional repertoire for some time and provided many examples of how they were used 
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in the classroom. In this context, it appears that a major conflict presented by the CBAAM 
centered on the appropriate use of pedagogical strategies in conjuction with specific, and in 
many cases, unfamiliar content. Many teachers talked generally about the alignment of the 
CBAAM with their instructional process, while others made specific references to problem 
solving, manipulatives, etc.. In addition, many teachers perceived that the types of items 
used on the CBAAM increased student involvement and satisfaction when doing 
mathematics. 
E2 

J: You said your students did well on the CBAAM cmd you attribute that to a 
lot of experience? 

E2: A lot of experience md experiencing success, md realizing that not getting 
it or making a mistake isn twhat it's shout It's the process. I teach process a 
lot. It's not winning, it's not getting it right, it's understanding. So I'm going 
for understanding all the time. You know, I'm constantly saying that I want to 
be sure that everyone understands. And they know. They aren't (fraid to say 
they don't understmd.,.because that's a natural place to be. The next step is to 
create the understanding. No one can learn including me...ifyou don't 
understmd what to do, or if you don't understand what it's all about. So the 
big thing to do is ask questions md find out. 

E7 

/ think it's [the CBAAM] improving the math instruction. Teachers are moving 
moie away from just having kids memorize or just learn how to do computation, 
to teaching kids to problem solve. The fact that it's on the test is motivating us 
to learn to have our kids.. .teach our kids to do better on the test But also, in the 
process realizing it's more fun far us to teach the way that the CBAAM is 
structured. I enjoy teaching math through the CBAAM much more than I enjoy 
it for the MATS far example. 

N9 

Several years ago I started using Marilyn Burns md problem-solving with 
mmipulatives...prior to the CBAAM. Sol was certainly in favor of the 
CBAAM because it did test much more closely to the curriculum that I was 
teaching. Well certainly the writing portion. How they solve the problem. We 
do a whole lot of that with my Marilyn Burns [Math Solutions] problem- 
solvuig, what they did md what they learned, md the step-by-step process that 
they went through. That lines up a lot The availdbility af mmipulatives. For 
example in the brownie problem. But the use of mmipulatives to solve the 
prdblems...yeak 

N10 

I've been doing this., for a long time. And I'm just thrilled that finally we 
have...* testing vehicle that is closer to what we're redly doing... .or some af us 
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are really doing in the classroom, and I think xve should all be doing it I just 
love it 

I got rid cf my textbooks three years ago, four years ago,..! forget. I just started 
going with the Marilyn Burns Math Solutions. So I think..the new tod (the 
CBAAM) is more in line with what I am doing in class. 

Extended invgiHgationa. Several teachers reported that they selected mathematical problems 
that required several days of activity to complete in preparation for the assessment. These 
extended investigations involved solving a real-life problem that requires completion of a 
number of steps en route to solution. This involved the application of mathemaH.al 
operations, problem solving approaches, and writing about and discussing mathematical 
thinking. 

E3 

£3: It gave the children the opportunity to show their ability to explain 
their thinking and explain the process they went through to solve the 
problem. 

I: Do you do that in your class on a regular basis? 

£3: Yes, I would say once a week Should be more than that, but at 
this point that's where I am. 

I: Is that part of what you would traditionally do in the third grade? 

£3; / emphasized it more. 

E5 

Well, I think it [the CBAAM] very much effects instruction in that we 
want the kids to do well on the test, or I at least do. And you know 
unless they've had this kind of problem-solving sort cf... situation 
or. ..practice they're not going to do as well as if they haven't 

E25 

/ think the CBAAM is opening the door to more investigative 
approaches to mathematics instruction. In grading the tests it's 
obvious that different people approach the teat in various ways; some 
teach to the test, others help very little. We need to emphasize the 
understanding in math - not just the rote skills. 

N8 

I: What part of your math instruction lines up with the CBAAM? 

N8: The idea that you do a math exploration and then you write about 
what you 've done. You talk with your partner about what things 
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you've discovered and look for patterns and things. Tying it to real world 
application alsa 

Cooperative Learning Strategics. Teachers have been using cooperative learning strategies as 
part of classroom instruction for years. The use of cooperative learning activities as part of 
an assessment process is a dramatically different approach than the use of traditional, 
standardized testing procedures. When traditional standardized testing is conducted, 
students work alone to solve multiple choice items and record their answers by filling in a 
bubble. Third grade students in Mt. Diablo classrooms are experienced participants in group 
activities. All the CBAAM problems were constructed to include group activities. Thus, 
teachers were encouraged by the nature of the CBAAM items to continue using cooperative 
groups. 

E2 

Well if you vxmt to talk in teaching styles.. .it used to be the old lecture. I never 
really was satisfied that way. 1 guess that's because I've been basically at the 
Elementary level I dm' t believe in it But now...we have individual projects, 
we work as partners, and we work in teams, cooperative learning. So that 
encompasses a lot of different learning styles for kids in wfccfc it's very helpful 
for them and it helps to prepare them for the future. But getting dam to the 
basics of the CBAAM, it's hands on. It's....prdblem*otving, it's critical 
thinking. I like tods to think 1 like to get tods thinking. Kidskveit They 
love it much more 1han...the person who stands and lectures in front of the room 
and does drill and alL.or even story problems out of a book 

E6 

J; Were the CBAAM items similar to the kind of things you do in class? 

E6: Yes, the scheduling not so much.., working with partners, working in 
cooperative groups,. .all that. .yes. 

Use of Manipulative!. The use of manipulates and hands-on instructional activities is 
regarded as an efficacious approach to mathematics and science instruction (NCTM, 1989; 
CDE, 1992). In the course of day-to-day instruction, Mt. Diablo third graders have access to 
many types of manipulatives (cuisinaire rods, counters, geoboards, physical models, etc). 
Many teachers reported that they relied much less on textbooks and some reported that they 
no longer used texts at all. In preparation for, and during the CBAAM assessment, students 
were also able to retrieve manipulatives as problem-solving aids although they were not 
provided by the district. 
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E5 

£5: Certainly the writing ivhich is related to the math and certainly the 
manipulative*, although you know it really galls me that they don't give you 
any mmipulatives to do the test. They just give you paper and pencil 

1: Do your kids get a lot of experience working with this in class? 

E5: Yes. About three times a week 

E6 

/ think we are now teaching in a different way. We are using a lot of 
mmipulatives. We do very UttU...just straight calculation. 

Writing Down Import a nt Mill. Many teachers encouraged students to give extended oral 
and written responses during instruction. However, several other teachers reported that their 
students received lower scores on *e CBAAM because they failed to provide sufficient detail 
to obtain a higher score. Some of th-se teachers attributed this to unclear administration 
procedures, while others suggested that students have not been taught, historically, to be 
thorough in their written explanations on assessments. Teachers also reported that in 
preparing for the assessment they chose activities that involved writing down the steps that a 
student uses to generate an answer. In addition, teachers frequently prompted students to 
write more as they examined student work during instruction. In general, teachers indicated 
that students knew what to expect in terms of the writing demands of the assessment. 
E7 

When the kids were given a task and asked to do some writing about if.. .my kids 
didn 't look bt me like...what do you mean? They know how to write about math 
or write about haw to solve a problem. So they knew what I was talking about, 
they weren't completely mystified when they saw the test Because we had done 
a lot of problems where they had t&..you know similar problems like that where 
they had to explain what their thinking was. My kids didn't actually do all that 
well on it but they didn't panic from it so I guess I made them feel more 
comfortable when they saw the test it wasn't like something they hadn't seen 
before. So I mink that emotionally they didn 't feel distressed by tin; test at all 

E19 

/ think the CBAAM gets better each year. The questions are getting clearer and 
students are better prepared for mis type of testing. Writing in math has become 
an integral part of our math program and tins has helped students approach 
these tasks with a positive attitude. My students liked the assessment and tins 
impressed me on the need for more good open-ended questions. A resource of 
good questions would be such a help as we know they are difficult to formulate. 
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inci-Msed Writing Burden. However, teachers were also concerned about the 
impact of the increased writing burden for students with poor English language 
skills and learning handicaps. In many cases teachers assisted these students by 
reading problems aloud and transcribing oral comments into written responses 
using "proxies". Teachers were in agreement that the CBAAM gives these 
students opportunities to perform on the problems, but that it is not a very good 
indicator of the students' mathematical capabilities. A few teachers were 
concerned about the writing burden, in general, and the stress that it places on the 
expression of mathematical ideas. 
E3 

What I have a problem with the CBAAM end with the CLAS test is...it's 
children of limited English proficiency and children who have special needs in 
regards to literacy. It doesn't tell you much at all 

E6 

The thing that I didn't do that I would probably do next time is take more 
dictation. I fust let Hum (ESL students) work their way through it and they 
used invented spelling. A lot of them have their initial consonants down so they 
will write that then put in a dash. So I would probably do more of that to make 
sure I have a good understanding of what it is they are trying to write. I sat 
dawn with four kids and asked what's this? And what's that? And tried to fill 
in the words for them. It's not a test of writing, it's a test of understanding. 

N28 

Take the letter writing, to anyone out of the test having students write a letter 
puts a whole different [level afi stress on the thought process assessed by Out test 
Just ask students to write! 

rio«» Alignment with Philo .nphy Advanced in New Mathematics Instruction. In addition to 
corresponding with new instructional strategies employed by teachers, many teachers 
described a relationship between the CBAAM and the "new" mathematics philosophy. In 
this light, cognitive psychological findings and constructivist philosophy have altered the 
way educators regard student learning. Increasingly, educators regard learning as an active 
process whereby students build knowledge structures based on networks of facts, principles, 
beliefs, and procedures. The construction of new knowledge requires sustained exposure to 
new content. Many opportunities to interact with the content through interactions with the 
teacher, other students, written materials, and manipulatives all contribute to the 
establishment of a rich database of information that a student can access when solving 
problems. The CBAAM items encouraged the use of instructional processes that help 
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students* construct knowledge that can be related and applied to their daily life. The 
emphasis on written expression helps establish links among the facts, principles and 
operational procedures that students must integrate in solving complex mathematical 
problems. Therefore, the assessment may serve an important communicative function; 
that conveys the nature of important mathematical learning. 
El 

It (the CBAAM) comes from the concept end of teaching.. .away from the 
C ™rtl atian 1* limutvr " I don't want to throw out the computation and the 
drill because there s some youngsters that learn the other way. They have to 
hm > a pattern to follow. So I think we still need the patterns, but it comes 
farther down the line after they understand, oh this is division. Then you sou 
ckay ths ts what the pattern looks like, and if you follow this it will work. 
That s the approach I ve used. Now I know I'm still putting some of the old 
vtew in with the new. But I really couldn't throw the baby out with the bath 
water. 

E3 

The district is very serious that we teach this way...and so that's a pretty 
powerful message they're giving with this kind of a test. So, with that in mind, 
that helped me understand, what we should be giving these children to do. 
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It does show., and I even heard some of those kids complaining to other kids, well 
it s not far that you guys get to do this and\..what you're learning is 
fun..Makmg comments like that. And sa..it makes a difference because the 
more positive experiences that kids have...in learning, and the more cooperative..., 
the more the teacher knows about teaching skills and learning techniques, and 
the more the teacher is willing to put in time and energy, I think the better the 
has will do because they re just going to have a better education. They 're going 
tobeastepup. And I also think that if the kids had a lot of experiences that 
were similar to what was going to be an the assessments...they're definitely 
going to do better on them. y 
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I think it certainly gives...teachers an idea about what kind of problems kids 
should be able to sohe...or the kind of problems kids should start to think 
about...Teaching them to solve problems in groups. I think it's helping them be 
™*' ****** ^ how they explain things, how they present things, and so I 
think they re just able to organize a paper better or organize their thoughts. 

I: But how is the CBAAM doing that? 

E7: WelL.just the fact that the problems are an the CBAAM, so we're doing 
more problems in the classroom. 
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It asse$ses...fir$t cf all can they solve a problem? Can they organize the 
information? Can they explain in writing, in a clear logical may to justify what 
they have done to get...to solve the problem. It helps teuhers because it shows 
them what the curriculum is supposed to look like. And you can't hide behind 
your textbook anymore. It's just not going to cut it 

Teachers Use of the CBAAM Results 

This section describes how the teachers used the results of the CBAAM. it was clear from 
the survey comments that some teachers, especially those opposed to the assessment, had no 
intention of using the information from the CBAAM. As the preceding sections indicate, the 
assessment takes considerable time and effort to prepare for, administer, and score. If 
information is valued and used to enhance teacher and student capacity to engage in 
challenging mathematical endeavors and enhance instructional opportunities, then the 
program may be worth the effort. If, on the other hand, information is not valued or used 
by the CBAAM participants, the benefits of the reform are limited to increased student 
opportunity before and during the assessment, but not subsequent to the assessment. 
Furthermore, written district guidance and workshops led by teacher development teams 
encouraged teachers and gave specific examples on how to use results to improve student 
performance and instructional programs. Nine of 11 teachers interviewed described a 
specific teacher or student use of results. 

Whole Class Instruction. According to teachers there was little consistency in the way that 
results were disseminated among teachers at the school-site. Teacher use was independent, 
without much collaboration across classrooms within a school or much direction or personal 
interaction on how to interpret or use the results. As such, results were used primarily to 
alter instruction within classrooms. However, a few teachers used the results to 
communicate with other teachers or parents. Teachers' enthusiasm for using the CBAAM 
results was mixed. Teachers who reported using the results, altered their group instruction 
and reflected on their own teaching based on their whole class' performance rather than 
using the CBAAM for purposes of individual diagnosis or student placement. Teacher use of 
the CBAAM results was consistent with the intended purpose of the assessment; the 
information was used primarily in classrooms as an instructional tool. 
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E3 

I, first of all, use it in my teaching as a tool to let me know how I'm doing. 
Where you know looking at these kids... where are they? Where do I take them 
from here? Where do the majority of the children score? What part of it were 
they lacking? You know, maybe they were Ale to come up with the right 
answer, hut they weren't Me to express it well 

E7 

Well, I looked at those (the CBAAM) in terms of how the kids did on the writing 
part of it And so on the report card, there's a section that says math and 
writing. And so I used.. .that a little bit ta..determine how the kids were doing 
in writing. Of course I had a lot of other examples.. .not just the CBAAM. 

N10 

What we have is a traveling portfolio that... we have pieces of writing in there in 
literature, and...the third grade has decided we're going put our CBAAM results 
in there. So the fourth grade teachers, some of them will understand where the 
children finished...at the end of the year so they can look dt what they did in 
third grade and take it from there into fourth grade. 

Articulating Performance Standards Via Rubrics. Many teachers employed the scoring rubrics 
as an instructional device to articulate performance standards during whole- class and small 
group activities. This was probably the most formal use of results and was outlined in 
written documents and at teacher workshops prior to the assessment. A few teachers 
indicated that the CBAAM and CLAS type rubrics have been integrated into daily activities 
for students. Teachers frequently reported the value of presenting the rubric to provide 
concrete examples to students of the types of responses that would receive a six versus a one. 
Some teachers reported using the CBAAM rubric to review student performance on the 
assessment. Other teachers suggested using the rubrics to explain student performance to 
parents when they visit the school. 
E4 

First, I went over it with them as a class, and 1 did it during early bird and late 
bird reading when I had the kids. I explained the rubric to them, and gave them 
examples of each one. And I told them.. J gave them the percentage at each 
level..! would say out of a hundred kids this is haw many kids got a four. Then 
we drew a graph so that they could see that. When they got their test back and 
they got a three, they could see that almost everybody else got a three too. Most 
of them were really happy about that. I think that redly answered a lot of their 
questions cbout...how come I didn 't get a six? S a.. that wot helpfuL.last year I 
don 't think we got that if I remember correctly... how many kids scored at each 
level 
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E5 

I: How did you use the CBAAM results? 

E5: 1 passed out the test booklets and...l gave them the...what's it called?. ..the 
rubric for each problem. Then rue talked about that, and of course this was over 
several days. 

I: So you went through each problem with the whole class using the rubric. 
E5: Exactly. 

I: Did you introduce the rubric earlier in the year? 

E5: No. Just as a result of this test and then giving some practice 
writmg...things that follow the same kind cf things. So that's why I have tins 
rubric, this particular one sitting here, so we went over that. 

E7 

Okay first I put it on the overhead...! made a simplified rubric...I didn't copy the 
one that was in the booklet because Out's real complex. I made a simplified 
rubric the kids would understand, a couple words for each one. I explained what 
the rubric meant. I passed back ...one at a time, or one booklet at a time. I had 
them look at their score and find out what their score meant. And then I put up 
an example of a six. What a six would look like for the "Hershey Bar" problem, 
and we went over it and talked about why it would be a six. And then I had the 
kids take a blank piece of paper and they practiced doing a six. Which they 
actually even had trouble with because some of them maybe if they didn't 
understand in the first place had trouble, so they just kind of copied what was on 
the overhead. Sol did have them physically say...you know I said to them, now 
if you could do the test over and get a better grade, what would you do to make 
your answer more clear? And so they all went through that The other two 1 
passed back and 1 explained the...let them look at their... lock at their score. At 
that point I did more modeling with the kids, where sometimes I feel like when I 
would...we would do an open-ended problem I'd correct it But a lot of times 1 
wouldn't go back and have closure and say to the kids, okay now you know this 
is what a...maybe this is what a real good one would be or something. 

N8 

I: Was that a result of the CBAAM experience? 

N8: The CBAAM and the CIAS botk..I decided that it's helpful for children to 
know what they're working towards. 1 also ordered the materials that were 
suggested in the CIAS manual They're open-ended questions and 1 use those 
as homework..acthnty once a week. And it gives a little checklist at the top for 
them ta..to kind of. ..it's a rubric type of thing to make that used pictures and 
drawings and given Hie multiple answers... there are those reminders. So they're 
getting familiar with that. We use a writing rubric alsa..in written work. One 
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of the teachers has just developed a science rubric.and some consistency in the 
form so that when you go from one...curriculum area to another. ..there's a 
slightly different rubric but it still has the same basic points to it. 

I: Are the kids adapting. Do they use them pretty well? 

N8: 1 think they like knowing. You know they can go back and look and say oh 
gosh I forgot to organize my work Maybe I II do Ms over ogam. 

N9 

/ shared the results with them and we have since made up our own little math 
rubric that they use in their notebooks... for grading purposes. It's only a four 
point rubric, but that's fine. Except for the "Carnival", we redly looked at 
basically here's what the sixes were. You know here is the best anybody could 
do, I shared that schedule with them, and we talked about what it was Out we 
attempted to do, and what the six was— they made it you know. Actually the 
six was a lot cleaner and easier than what they had tried to da They just tried 
to do too many things. 

NIO 

NIO: Well, there's good and bad to that because what I've done is to xerox off a 
copy of. ..what O...I think there weren 't even that many sixes. The children 
compared what they had done against the exemplar. And then, we discussed 
well how we cm improve. Their job was to take one cf the problems and to see 
if they could improve it...to come up one.. .and most af them did. 

I: Were you happy with that? 

NIO: I think if you use it as a valuable learning tool to take a look at where you 
are...the student is going to be able to do it...self-evaluate...then it's valuable. 

I: It lent itself to that? 

NIO: Oh yeah, but all of their...writing and rewriting lends itself to that. We do 
it in science, we do it in math.. .we do it in.Janguage arts. 



Mot i vating Students tp Improve Writing . Some teachers used the CBAAM and the rubrics 
a springboard to motivate students to improve their writing. The rubrics gave students an 
opportunity to see the amount of detail and quality of writing required for a high score. 
E6 

/ know it's [the CBAAM] changing the way...not so much the way I'm 
teaching because I've always used a lot of manipulatives and things 
like that.. .but in writing and explaining.., we need to keep looking at 
that. Explaining themselves in writmg...explaming fair thinking and 
how they're thinking. They're not very good at that and they haven't 
had to do it until the third grade. I expect it all the time. We write 
about math everyday. 



45 



51 



N8 

That's the way.. .they see themselves. Anything less than a six is 
devastating and to get a three instead cf a five...was hard. In fact, I 
gave them a chance to go back md rewrite anyone of the items that 
they wanted. I said take this as a sample md now let's see if you can 
do abetter job. I gave back their tests. I explained to them how they 
were scored, what the numbers memt And just in general terms that 
if you got a four, five or six you did well, and if you got a three or 
lower then you need to make sure that you write it so that somebody 
who's reading it.J didn't want them to think that they had failed, but 
that they had failed to write down everything that they knew. 

Communicating Standards and Ngw Instructional Approaches to Parents. Some teachers 
described plans to use the CBAAM results to communicate the nature of the new 
mathematics instruction to parents. The types of items used on the CBAAM , as well as the 
scoring rubric, would give parents a clear example of what their children were expected to 
demonstrate in contemporary mathematics classes. Teachers reported that parents still 
expected their children to bring home computation worksheets. When parents get exposed 
to the actual CBAAM items they are better able to understand the types of instruction and 
homework to which students are exposed. However, in many cases teachers indicated that 
schools haven't done the necessary ground work with parents during this transitional period. 
El 

When you explained what the grade meant. .if the teacher was happy with 
it... and I let them htow that I was...they were fine. And I didn't have one bad 
comment from a parent 

E2 

That's another thing that I've been working with the Principal on. 
Suggesting that what we need to do is...to provide education for our 
parents. Now Ms year we only had twenty minutes.. .for back to 
school night, md you know, I like to talk m hour md a half.. J have a 
very involved program. So I suggest that we go back... take Hie parents 
through some of these experiences, have them do it so they cm 
understand what's going art md how it applies. 

E3 

/ just think that's mother way that the district's going in the right direction. 
When parents start seeing that tests are being given this way they an much 
more open to the idea that there aren't as many worksheets coming home. 

And alsa..you're playing a lot cf games md my kids don't know the math facts. 
You get questions in those regards md when they see a test like this then they 
cm begin to understmd. 
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N10 

Tlte kind of person that works here, first of all is very dedicated and 
very innovative. We are interested in doing the best job with the 
curriculum that we can based upon research, and to evaluate what 
we're doing. Then see if it works, and then reevaluate, and go forward 
and see if that works. The mistake we made is that because toe were 
still . .forming... the concept of the portfolio and the rubrics and 
everything, we did not include enough of the community in the 
education process. So you know a lot of parents said tar// what does 
this mean? So we know now I think for next fall, tins coming fall '94, 
that we need to have a specific night where we will have the exemplars 
out and educate parents.. .so that they understand what a six is. What 
does that mean? What does that entail? 

A few teachers purposefully avoided active use of the CBAAM results with their students or 
sharing remits with parents because of concerns over the effects of what they perceived to be 
low scores, on student motivation. In one case, the teacher selectively reviewed items with 
students (r viewed "Hershey Bar", but not "Carnival"), while another teacher gave no 
feedback or individual student scores. These teachers indicated that students worked hard 
on the asse> > nent items and would likely be discouraged with the results. 



Contrast of Traditional Tests and the CBAAM 

Emergence o the MAT-6 Themg. During the interviews, many teachers from both involved 
and non-invo ed groups commented on their dislike of the MAT-6 test, both its format and 
content. This vas not a topic generated by the interviewer, but usually came up during the 
discussion of i ie CBAAM alignment with instruction or use of test results. In many cases it 
arose as a contrast to the CBAAM design and subsequent teacher use. In others the 
interviews took place near the MAT-6 testing dates and arose when discussing current 
activities. At that point the interviewer began to probe whether there were specific uses for 
the MAT-6 results. Only two teachers reported use of MAT-6 results. After probing about 
possible uses the teachers began to explain why they failed to use the MAT-6 results. 
E4 

7; You mentioned the MAT-6 results. What do you do with them when you get 
them back? 

E4: Well, I stick them in the cumulative files end I send the results home . 
to the parents. And I worry about some fads that have really low scores. 
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/: Do you do anything with the scores and then make changes or...? 

£4; Na And I think that the actual test is upsetting for a lot of the 
kids.. .they 're not used to working that way for one. Tomorrow 
we* retaking the practice test and I'm going to get them ready. And tell 
me, we're not going to be working together, we're not going to he asking 
questions of each other. And I'm going to do it during my reading 
period because I feel that..that way lean at least give them some peace 
and quiet. And they all don't all have to sit next to one another. 

I: It's quite a bit different than the CBAAM. 

E4: They really enjoyed the CBAAM. Most of them are net going to enjoy the 
MAT-6. There '$ a different kind of pressure. It's a test It's not an assessment 
as far as they're concerned. It's a test It goes in their file <md people lock at 
their scores. And they're eligible for certain things according to their scores. 

E5 

£5: Not really because especially the MAT-6 is so far out of kilter as far as what 
we are doing now in the classroom. 

I: It is? 

£5; Yeah, it really is. 
I: In what respect? 

£5: Well it's still basically paper mid pencil math, cexd what do you understand 
about this problem? Or...what is 300 + 25? What is 300 + 20 + 5, and...those 
kind of things we don 't do much mtymore. 

I: Do you think the CBAAM and the MAT-6 together would give you good 
information? 

£5; Definitely. I mean certainly the CBAAM is a lot more curriculum-based or 
more relevtmt to our curriculum today <md the State Framework cexd so forth. 
Although I think there is still a place for the MAT-6 even though it's pretty 
much out of date... curriculum wise. There's still a place for it I think in that it 
gives you a more detailed idea cf where the children are now, cexd what maybe 
they don 't know in specific skill areas. 

E6 

£6: They don't do very well on the MAT-6. They don't perform anywhere near 
as well as they do on the CBAAM. We end up being down in the 23rd 
percentile on a lot of those. 

I: Why do you think that is? 
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E6: Because I think we ate teaching in a different way. We are using a lot of 
manipulative*. We da..very little, just straight calculations. And...their math is 
their best. I mean generally they are more competitive in math than anything 
else. More than reading or anything like that. 

I: What do you do with your MAT-6 results when you get them hack? 

E6: Trash them. I don't think they're valid. J don't think they're that 
representative of what the kids are doing. I think the CBAAM is much more 
representative of what they're doing. If I want to talk to a parent, I have to say 
your kids in the 23rd percentile and you know that means...76% of kids scored 
higher than they did. But if I can say they have the computation for the 
multiplication, but they don't exactly understand the concept cf it..., they need 
to have a better picture of what their kids look like. The parents will work on 
that, but it's different than what the parents are used to. 

N9 

N9: Vm definitely in favor of the CBAAM as opposed to I just gave the kids the 
MATS. Well the MAT-6 now is so foreign to the xvay I teack I can't speak 
for somebody else, but its really foreign to the way I teach Where the CBAAM 
is certainly in line with what I teach The concept cf the CBAAM is definitely 
in line with what I teach And for myself I give it more credence.. .than I do the 
MATS. 

I: What do you do with your MAT-6 results when they come back? 

N9: 1 think they send them home. They use them for determining gifted and 
talented. 

I: Do you use them to make any decisions? 

N9: No, because its just so foreign to what to what I do now. Multiple choice 
is so... 

I: So... if doesn't give you the kind cf feedback you need about what you're doing 
in the classroom. 

N9: No, no. And it certainly wouldn't lead me...to teach. .to do that either. 
You know 1 just don't find it as meaningful as I used to. And.. J don't worry 
about the results now. I used to worry about the results. I don't. It doesn't go 
with my curriculum so... 

N10 

N10: The MAT-6 is nothing. It is a waste cf money. It's a waste cf time and 
just freaks the kids out to all cf a sudden have to... 

I: Why do you think it freaks them out? 
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N10: It freaks them out because we dan t do a whole lot of just straight 
computational kinds of questions that they say divide into this. It has nothing to 
do with what they're doing in class. 

I: Do you ever use those kinds of formats, Jihe the MAT-6? 

mO: I don't I don't use that format at all We d*..atl cf our math writing in 
a Matk..notebook They do all their problem solving and all the writing to 
explain you knew what they've done. We keep everything there. 

I: Well, what do you do when you get the results back at the end of the 
year? What do you do with those results, with the MAT-6? 

N10: These go home..MAT-6 results go home to the parents. 

I: So what do you do with the teacher results? 

N10: Throw them out. 

I; Do you look at them and change instruction as a result? 
N10: No. 

I: Do you evaluate what skills your kids are doing well on as opposed to not so 
welL.with the MAT-6 results? 

N10: Well, I don't...! don't hold much weight in the MAT4 at all I know the 
parents da And so they want to know what liappened here and why didn't my 
hd do well? I say well that's not the way I teack..*uL.Mnfortunately the 
assessment tool does not match the curriculum nor the innovations that have 
been made as far as learning and how to explain. 

Instructional Alignment. According to teachers, the format of the CBAAM was more aligned 
with typical classroom activities. In addition, teachers are making less use of highly 
structured materials, including problem sets, text books, and multiple choice formats. This 
makes the CBAAM quite valuable in communicating essential content and constructed 
response problem format among teachers, parents, and students. It should be noted that the 
district encouraged teachers to discuss the CBAAM results with parents, but not to send home 
scores or papers without some formal communication. Several teachers planned to include 
the CBAAM in parent conferences where the mathematics instruction and the CBAAM 
results could be described. 
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When discussing the CBAAM results, the teachers recognized that the CBAAM does not 
thoroughly assess computational skills or algorithms taught in the primary grades. A few 
teachers indicated that a combination of the MAT-6 and the CBAAM results would provide 
the best picture of student performance. However, because of the perceived misalignment 
between the MAT-6 and classroom instruction most interviewed teachers limited use of 
results to "mandatory" requirements like placing them in the student folder, sending test 
scores to parents, and using them to identify gifted or Chapter 1 students. Only one teacher 
described analysis of whole-class results and specific linkages to classroom instruction. Based 
on teacher comments, the CBAAM seems to better serve teaching /instructional functions, 
while the standardized norm-referenced test serves primarily administrative functions. 

Articulation/ Alignment of the CBAAM 

Most teachers interviewed in this study, saw close alignment between the CBAAM and 
classroom process. They see the CBAAM as measuring important mathematical ideas. They 
have concerns about the specific content tapped in the CBAAM problems, yet, they changed 
their instructional sequence and curriculum content to accommodate the assessment and they 
did use the CBAAM results within their individual classrooms. This next section of the 
report discusses how the CBAAM is one mechanism that fosters articulation of content 
standards at the district, school, and classroom levels. The CBAAM fosters articulation of the 
mathematics content standards by illustrating complex, open-ended activities. The 
assessment provides opportunities for teachers and students to engage in tasks similar to 
CLAS type items. This is particularly important given that the interviews provided evidence 
that teachers have limited opportunities to discuss the new curriculum frameworks and 
coordinate and communicate between and across grade levels. There was some evidence that 
teachers at some sites were collaborating with their peers on the mathematics program. 
These efforts, however, appeared self-initiated and were not systemic in nature. In this 
context, the CBAAM, its development, administration, scoring and review of results become 
an important vehicle for articulating important content and illustrating what performance 
looks like. 
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Within and Between Grade Articulation The CBAAM is one step in preparing students and 
teachers for what is expected by the new frameworks and what will be measured by state 
assessment. The interviews provided evidence that within grade levels there is considerable 
variation in the amount of discussion and interaction among teachers about mathematics 
curriculum and specific content. The following quotes illustrate how the within grade 
articulation is dependent on the school-site, or individual teacher. 
N9 

Our program here.. .at least once a month we have grade level meetings tvith just 
our own colleagues. There 's another third grade teacher and I who initially took 
the Marilyn Burns Math Program and started it It makes a whole lot of 
difference. We all know where we are and what we're doing. And what our 
goals are, even though we may do it in a different way, we're all in Out same 
track. 

Nil 

/ think it's horrible. We don't, at this site, do any grade level planning together. 
We don 't..do any...collaboration among teachers here. So it doesn t happen 
for..l mean a number of reasons. But...personalities...although it's not a 
personality thing. It's just a matter of allocating the time to do it 

There was less variation in cross-grade articulation, but the interviews suggested that it is 
generally weak and less formal. However, there is some evidence that it is beginning to 
occur. Teachers reported awareness of the need for coordination across grades. The 
articulation occurs through informal interactions among teachers from adjacent grades and 
the sharing of materials and assessment tasks. Teacher comments indicate a pervasive desire 
for better communication and coordination of mathematics programs across grade levels. 
There is a strong sense that this is necessary to prepare students for the new challenges that 
face them. Teachers recognize that success on new assessments and the curriculum that 
coincides with them, speaks against maintaining cultures of isolation (Jamentz, 1993). 
ElO 

I'm seeing some of the changes. And I think basically a lot of the changes start 
in first grade and kind of work their way up. But I think they need to be more 
organized. More feedback between the grade levels. And we have tried here to 
do that. But in order to do it... it either has to be before school, after school or... 

E3 

/ think that's what the other teachers found too...is that if the kids are not there, 
and even though you go over to them and show them what a three, four or five, 
should look like it doesn't do them a darn bit of good if they're not there. And I 
think getting there means starting it kindergarten... 
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E4 

Well, one of the things that I really enjoyed was when we had the staff 
development day for the CBAAM. They gave us packets of things that we could 
take and try immediately in the classroom. Another tiring that would be helpful 
is to provide teachers at every grade level with examples of the kinds of tilings 
that they could, or should he doing? Not only to prepare the children for these 
kinds of assessments, but I have heard some of the teachers saying.. .in first grade 
or second grade, well what kinds of things should we be doing. A lot of the 
things...the kids should already know. They should already have some 
experiences...and nat...we get the test in two weeks, we have to teach them 
everything that they're supposed to know. Because it's just not going to happen. 
They need a lot more time to get used to the ideas. 

E6 

That's my sense of it. And we're getting no support at all in first and second 
grade levels to get some more writing built-in and to get the discussion of math 
concepts built-in, because they're obviously very shocked. That's real unfair to 
the CBAAM as a pre.Jfit's intended to be a precursor to CIAS. 

E7 

To tell you the truth I'm just not sure about the lower grades. Just.. .being new 
here... I haven't had a lot of interaction. We have different lunches, different 
recesses, different everything. So I just feel like at our school the upper grade 
teachers, which includes me too, the third grade and up, really don't do a lot of 
discussion with kindergarten or second grade teachers. I don't think you can 
expect third graders to all of a sudden do really well with the CBAAM if they 
have not seen it before. So I think they should definitely be starting off in 
kindergarten. Obviously kindergarteners aren't going to be writing, explaining, 
and thinking on paper. But talking about it in class and maybe having a teacher 
doing writing for them and having them see a lot of it (might help). 

E40 

The CBAAM problems are better, but need to include quality models (examples) 
for students - so tine child can see what to strive for. The project has helped to 
chmge curriculum and instruction and should be brought down to second grade 
and below Field testing with a variety of classrooms would insure we are 
assessing what is intended 

N8 

There's an attempt to do some cross-grade level articulation. It hasn't happened 
yet, but it's been written into the plan for next year. ..that it be done on a formal 
basis. I have found...anytime I've gone and asked, I've found the support there 
very strong. But it's a matter of going and looking for it becaust you don't 
have the opportunity. 
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N9 

There's been a change in second grade teachers.. .and particularly in the last 
couple of years. Far more of them have gotten on this track Which makes it 
much easier. The second grades are doing a lot more and so are the first grades. 
So it's working its way down. 

N10 

It 's very.. .concept building. It has to start from kindergarten in approaching 
these problems. You cannot just att of a sudden get to the third grade and say 
hey they're ready because I've been teaching Ms way the whole year. It starts 
in kindergarten and it builds... 

The CBAAM Become a Vehicle far Articulating t he Framework. It is unlikely that the 
curriculum framework itself will significantly impact teaching and learning in Mt. Diablo 
classrooms. Most of the teachers interviewed were aware of the framework, had read 
portions of it, but did not use it as a foundation for their instruction. The following excerpts 
reveal the perspective that frameworks are valuable guidance, but of limited utility for 
altering practice. 
El 

I: Do you get a sense that teachers are using the 1992 Frameworks as guidance? 

El: Do you want the truth? That's not reading that you sit down and enjoy...at the 
meetings in math. We took a whole day.. .many teachers in the district, and did a bird's eye 
view of the framework That's what I presented to the rest of the faculty here,..is the bird's 
eye view because I know they're not going to sit down and read it unless somebody makes 
them. There are too many things to do. You know, if can you just keep your head above 
water, and survive, and get the papers graded, and get tomorrow's work ready and 
everythmg...and the last thing you want to do is to sit down and read a math framework 

E5 

£5: Certainly...being an old timer I wouldn't of been that eager to get into it..if there hadn't 
been. ..the CBAAM as a way of evaluating how your children are doing in the so-called new 
math. 

I: Have you read the framework, the '92 Framework? 

£5: No,...although it would be a great way to go to sleep, probably. I've read parts of it at 
different times.. .but only.. .here and there. I've never actually sat down and read the whole 
thing. 

E7 

The new framework..! haven't read it completely. I've gotten a brochure on it that gave me 
an overview of it but I haven't read the framework I'm familiar with the framework 
previous to that which was...I guess...kind of similar. ..is what I've been told. There's some 
changes, but in general it seems like it's kind of on the same track 
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N9 

I think in as much as it's (Framework) helped to keep me on track I mean in other words it 
also verifies what you're doing. Since the framework, the new framework does deal so much 
in prdblem-sotving math. It s a tod to keep you on track A guide. And I use it that way. 

N43 

J feel the CBAAM helps teachers who are hesitant in trying new techniques to get 
going on the new framework I feel that the CBAAM was in-line with what I 
have been teaching in class. I would like to give the test in March instead of 
January. 

Other intermediate activities, such as informal grade-level meetings and CBAAM activities, 
translate and bring to life the ideas in the framework. In a similar way, formal participation 
at inservices such as "Marilyn Burns workshops' 1 may be one of the mechanisms for 
translating the ideas in the mathematics framework into classroom practice. Given that 
teachers report using a less standardized curriculum in their classrooms, it is critical for 
assessments, of any type, to communicate what is important and valuable mathematics. 
From both the teacher and district perspective this can help educators identify, select and use 
relevant instructional strategies and materials. If teachers rely heavily on texts to sequence 
and structure their classroom instruction, they may be less reliant on mechanisms, such as 
the CBAAM, to determine direction for their programs. This interview data suggest that 
teachers are moving away from classrooms that are structured by texts, therefore it is 
incumbent upon the district to provide tangible linkages from the state level frameworks to 
classroom activities. 

During the CBAAM item development and at the district-wide inservices for third grade 
teachers, explicit linkages were drawn between CBAAM items and the unifying ideas and 
content strands in the 1992 Framework. However, in contrast to this, several teachers 
commented on the lack of alignment between CBAAM content and district-level mathematics 
guidelines. Although they recognized that the district guidelines were published prior to the 
framework, they were concerned about the different messages that various components of 
the mathematics education system are sending to teachers, parents, and students. In this 
period of change, teachers are struggling to maintain a balanced approach to mathematics 
curriculum and instruction and there is a strong sense that explicit guidelines and major 
assessments should be consistent and reflect that balance. Forging the balance involves 
integrating basic computational skUls with challenging problem solving and investigative 
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approaches, as well as, synthesizing new content and pedagogy in ways that are meaningful 
for students. The CBAAM is not just an informal assessment; it's a statement about 
important mathematics. Teachers want these statements - curriculum frameworks, district 
curriculum goals, lie CBAAM, and other assessments - to be aligned. 

Creating and sustaining a challenging mathematics program involves much more than 
revising district guidelines or discussing state curriculum frameworks. At the heart of the 
matter is the strength and success of a professional development program that enhances the 
capacity of teachers to engage students in challenging content. The scope of the CBAAM 
project (it has affected all third grade teachers for four years) provided an opportunity to 
relate teachers perceptions about professional development in mathematics to the CBAAM 
project. 

Professional Development Directed Towards the Mathematics Program 

In general, professional development in mathematics comes from a combination of sources, 
including: preservice training; inservice opportunities within and outside the district; 
interactions with students, peers, and other educators; and individual endeavors to develop 
competence. The CBAAM is an example of a district initiative that supports the 
revitalization of mathematics programs and links this to the ideas set forth in the NCTM 
standards and the California curriculum frameworks. In this context, working on the 
CBAAM development teams, administering and scoring the assessment, and interpreting 
student results can be viewed as a substantial professional development activity for 
participating teachers. 

Structured opportunities for professional development in mathematics are particularly 
important when considering that elementary teachers generally have little formal training in 
mathematics. Typically, they take one or two courses in mathematics content and a course in 
teaching methods. It is not unusual to find teachers who have no specific mathematics 
Gaining, because elementary preservice programs have historically been laden with reading, 
literacy, and language arts coursework. This makes ongoing professional development 
programs in mathematics all the more critical. 
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El 

The minimum. I went up through..algebra and geometry. Did okay there, but 
when they came to calculus and stuff, I went to music. But music is nothing 
but math with sounds to it. 

E6 

Mas* of us can muck our way through addition and subtraction without a lot of 
help, but when you get into broadening math beyond aritkmetic.into the whole 
scope of it, you have to know some physics and you have to know some 
geometry, and some algebra, and all that...I think we're probably weak 

N8 

The Elementary Mathematics is simple to understand from the kind of 
background I had. If I were going into teaching at the High School I would need 
a lot more training of course. But...actualhj, I don't really feel that in my 
credential program they did a good job of snowing me how to teach 
children...how to teach math. Since then I've been in the district and I've gone 
to lots of workshops. The district has had a lot of in-service training that has 
helped me leant. ..helped me learn to teach math better. 

Framework Consistent Intermedial AfHviH^ in their research on the impact of the 1985 
Framework, Cohen et. al. (1990) expressed a major dilemma posed by mathematics reforms: 
"How can teachers teach a mathematics that they never learned, in ways that they never 
experienced?" (pg. 6). They explored the complexity of large scale curricular reform and the 
difficulty of producing change driven by state policy. The primary difficulties were not in 
accepting the new policies, but weaving them into daily practice. They found that teachers 
implemented many elements of the frameworks, both content and pedagogy, but tended to 
organize these new elements within the structure of traditional school mathematics. 
Implementation of the new ideas was "filtered" through the older and more traditional 
mathematical and pedagogical structures. Thus, teachers had much to learn, and unlearn in 
order to translate the frameworks into practice. 

Research findings pointing out the piecemeal adaption of "new" mathematics practices by 
teachers may be due, in part, to the piecemeal nature of support and professional 
development that they receive on their journey. Given the magnitude of changes proposed 
for elementary mathematics teachers, it follows that enormous resources are required to put 
fundamentally new practices in place. Yet, school systems continue to rely on college courses 
and short-term in-service programs when the demands of new reforms call for long-term 
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professional development activities that engage teachers in new learnings and the 
redefinition of practice over many years (Little, 1993). These long-term experiences are 
necessary to translate the challenging ideas in the frameworks into meaningful classroom 
actions by teachers and powerful new learnings for students. In the discussions with Mt 
Diablo teachers, they tended to view the CBAAM as a translation activity that helps 
articulate the ideas in the framework. 
N8 

My understanding is that there's been a lot of growth in 50% of the third 
grade...teachers moving from a more traditional teaching style to some of the 
newer...approaches. And I fhmk the CBAAM has fad an impact Because I 
think it gets right down into the classroom, whereas if it's just a framework that 
sits on your desk, you may never actually bring fluxe.. theories into practice. 

Teacher Perceptions of Formal In-service O pportunities. Formal mathematics in-service 
opportunities have also been available through the district. The interview data suggest that 
although a variety of in-service has been available through the district, teachers expressed 
concern that they are not a long-term systemically oriented program. Nor do they provide 
in-depth treatment of a central mathematical idea in relationship to the essential strategies 
required for successful instruction. For example, most teachers reported attending in-services 
provided by Marilyn Burns, or based on materials or videos authored by Marilyn Burns. 
Some of these in-services were sponsored by Mt. Diablo, others were outside the district as 
part of summer retreats or national workshops. For teachers what makes the in-service 
attractive is that it does not focus simply on mastery of mathematical ideas, but serves as a 
basis for the translation of these ideas into accessible instructional strategies. Many of the 
mathematical ideas presented in the framework are complex and represent new classroom 
content. Similarly, the CBAAM helps teachers to illustrate and translate some of these ideas 
into classroom practice. Some Mt. Diablo teachers report ample opportunity for their own 
professional development. Others were not aware of these opportunities and still others, 
were aware, but opted to not participate. The interview data demonstrate that opportunities 
are available when teachers take it upon themselves to identify and participate in training. 
However, there is a strong sense among teachers that the district support does not present a 
coherent, long-term strategy for upgrading the mathematics program. They see the CBAAM 
as one link in the chain, but often stressed the need for more opportunities to work together 
in both formal and informal ways. 
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Informal O pportunities. Information from a formal in-service or the CBAAM administration 
may only represent the first step in the cycle of collaboration necessary to enhance one s 
capability to engage in teaching practices demanded by the new reforms. The next steps 
require applying newly acquired knowledge, sharing successful strategies with colleagues, 
and reflecting on one's own practice. These reflections enable teachers to refine and improve 
their pedagogy. It is these changes in teaching that sustain new opportunities for students. 
Dialogue and collaboration are the informal opportunities that can buttress these new 
learnings. 



Interview data demonstrate the desire for increased interaction within and across school-sites 
to accelerate the substitution of new practices for old. Teachers expressed a need to work 
with each other to implement new ideas and not have external consultants or experts define 
the pathways to success. Without adequate opportunity for collaboration, the changes will 
be spotty and not reflect a fundamental transition from a lock-step curriculum centered on 
sequential computational skill acquisition to one balanced with teaching for conceptual 
understanding. Based on the interview data, Mt. Diablo third grade teachers are clearly in 
the midst of this transition. Yet, they are struggling to make sense of the complexities of the 
mathematics reforms with little time and few resources. 
El 

I've had one opportunity in my 20-something, well next to 30-years in the 
district. One cf our staff development days toe did a trade where we went to 
different schools.. .and watched a program. I saw something I liked, picked it up, 
and have used it ever since. So I think it would be good One tiling that 
disturbs me is the new teachers coming out of college.. .that are being stuck into a 
classroom with no help. 

E2 

/ think even if they did [collaboration]. ..kindergarten, first grade together, and 
then have them break into groups and come back and talk to one another.. .a day 
or two days. Second, third or. ..that kind cf thing... going up. What bothers me 
so much about this is you don't see an awful lot of it. 

What we're doing is good but we need to do more cf it. And I think we need to 
do more cf it in schools, I think there needs to be more from the principals. I 
think there needs to be more cf an in-house look Not pointed, but making the 
stuff available for people. Chances for people to team up and talk and be in 
teems just like we do with the kids. 
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E3 

I: Are teachers involved in professional development on a regular basis? Is the 
district going through some staff development in math? 

E3: I think that's their goal, but mere's no blueprint right now. There's nothing 
to get your hands on at this moment. No. 

E4 

Well, I mink it would be helpful to...maybe some teachers won't do it. 
It's the same thing like with the Other third grade teacher that..was 
supposed to go to the CBAAM in-service and then didn't Sol picked 
up all of the information and gave it to the teacher and gave all the 
examples of the tilings that I was doing, but I realty don't think that the 
teacher did any of mem. I realty don't Maybe he did but...l don t think 
so. And sol think that it might be helpful to have some kind of mam 
fair for each grade level I know I've seen some school* that do that, and 
each class would be given the opportunity to work on a certain type of 
problem and then they would present in a multi-use room or something 
like mat, all of their solutions. 

E5 

We're lucky in that we all like each other and want to... work with each other on 
that kind of thing (math). It's a good group...we always eat lunch together and 
we're always talking about things... inhere we should be next and mat kind of 
thing, and we have a shortened day on Wednesday so mat every fourth 
Wednesday is a so-called...well every third wednesday...is a so-called grade level 
meeting. And then we talk about things. 

E7 

My understanding is that mere ham 't been a new math adoption for quite 
sometime. And so all the math books mat we have mat the school adopted are 
all kind of.t.there really isn't any open-ended questions in the mam book There 
are some word problems, but most of it is just oU-fashioned...what I would 
consider old-fashioned math. And so I feel if I was given a unit. .you know 
units to work on, I would teach it more. But I feeL.a lot of teachers are really 
frustrated because here they're having to scrape for materials and to get problems 
from mis teacher, and get a book from this teacher and it. .almost seems like it's 
piecemeal I mean, that's how I feel I've bought three books myself.. Marilyn 
Burns books...I bought a replacement unit for Marilyn Burns but, I just feel I'm 
pulling things from here and there. Whereas, I have a good friend who's teaching 
in another district and there the principal has bought a.. .unit A mam unit that 
she's piloting, and I looked at it a little bit, and realhj...it's not a textbook It's a 
big book It's a binder. 

N8 

/ would like to see more sharing. I think there are a lot of teachers with talent 
and knowledge mat don't share back and form. Within our school we're pretty 
good about that, the way we have our meetings setup. We meet at least once a 
month formally as a third grade. And we do work closely with one another. 
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But I ivould like to know what's going on in other schools. I had an 
opportunity to visit some other schools as a result of a proposal that I wrote. I 
spent some time in three or four other third grade classrooms. And 1 found it 
very valuable. And I think that kind of thing is good. I think anytime you can 
get together and just share what you're doing in your classroom...you bounce 
ideas off of each other. 

N9 

Well, assuming that they're holding inservices. As I say some of that stuff 
comes across my desk and I don't even look at it. I don't know. Maybe they 
could set up a visitation program? Visit classes where it's working and it's 
working okay, and see if they can't help people become more comfortable about it. 
There's a lot of people 1hat...are just afraid of change. 

N10 

Teachers need release time. A list of people that are really into hands-on math 
and writing in math and integrating the curriculum. Teachers need to have, 
even once a month, or twice a month, some kind of release time to go and visit 
and see, and then have a support group that you am come back and share...what 
worked for you, what didn 't work for you. I had mis problem...somebody helped 
me solve it. That's what's going to make people really feel comfortable enough 
to own it, to feel excited about it, to move forward with it. Without that support 
they're going to just say here, mis is what CLAS is. You're not going to get 
anything. I think most people on-site here are feeling more and more comfortable 
because we do communicate with each other. But if you're talking across the 
district.,.1 think the district needs to make a commitment. The district needs to 
make a commitment to supporting teachers and to educating teachers and the 
community. 

Nil 

J think there was one on probability. But, it wasn't taught by teachers. I mean 
it needs to be. I know what probability is, and I can go learn what probability 
is. I mean there is enough books out. .you can just get a book on probability. 
But it needs to be third grade people teaching these third grade skills, and 
coming up with some neat ideas and some good ways to do it mat works. 

The CBAAM aa a Formal Professional Development Activity. Teachers discussed a wide 
array of professional development activities, but the CBAAM initiative was seen as the only 
district-wide, long-term effort. Some teachers, even those making substantial contributions to 
the project mentioned that the CBAAM effort fell far short of what is required to make the 
changes proposed by the framework. However, some teachers have participated heavily in 
CBAAM related activities. They have constructed items, administered the assessment, and 
scored and reported results; while others have only administered the assessment in their own 
classroom*. The level of involvement mediates the benefit to teachers and students. This 
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research does not credit the CBAAM as primary cause of recent changes, but identifies its 
critical role in supporting difficult and time consuming transitions for l?*chers. It does this 
primarily through communicating, combining, and modeling new mathematical ideas, 
content, and instructional strategies and sanctioning them in the classroom. Based on the 
survey and interview results, a large majority of teachers believe that the CBAAM models 
important ideas and strategies or contributes to the mathematics program and student 
learning. In most cases, teachers' attitudes toward the CBAAM were positive, but in many 
cases this type of assessment and the related instruction represents a departure from 
"business as usual" and places demands on teachers that make the transition difficult. Even 
if a teacher is positively disposed to the philosophy underlying the CBAAM, the increased 
demands for planning the requisite changes necessitate district sanctions for formal and 
informal professional development beyond teacher participation in the CBAAM assessment 
project. 

Teacher Concerns About the CBAAM Administration and Procedures 

It was apparent during the interviews that many teachers were not clear about the specific 
purposes of the CBAAM project, how the assessment information was to be used or not used 
by the district, and the saliency of specific administration procedures. Most concerns 
involved uncertainties about the administrative procedures they followed when giving the 
assessment. The CBAAM procedures extended over several days, involved group activities 
and permitted the use of manipulatives and other classroom materials. The pronounced 
differences between this type of administration and standardized assessment procedures was 
the source of much concern by teachers. Some teachers were unclear about the amount of 
help they could provide to individual students. They wondered if it were appropriate to 
repeatedly prompt students to write more or provide additional detail in their answers. 
Other teachers reported difficulties with students who could not easily understand the 
CBAAM format. Some special needs students found the instructions too complicated and 
were unable to begin the assessment. Some teachers reported that LEP students found the 
writing burden of the assessment excessive. Teachers commented that often their highest 
performing students' found it challenging to generate problem solutions and write with 
clarity and supporting detail. By extension, they believed that low performing students 
would experience even more difficulty in presenting ideas they may understand but do not 
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express well in writing. Teachers differed in the amount of time they permitted students to 
work on the assessment. This time variation caused many teachers to question the validity of 
the results, given that students experienced different administration procedures. During the 
administration, some teachers had posters and materials on display in their classrooms that 
might have facilitated answering the CBAAM problems. Several teachers questioned the 
appropriateness of this practice. Following are just a few examples of these concerns: 
E4 

They were allowed to get up and walk around, and they were allowed to use 
color tiles if they wanted to. They were allowed to use construction paper. I 
told them they could use any of those things if they wanted to. I didn't go as 
far as one teacher did, which was to provide them with construction paper... with 
little dotted lines on it. And that... I thought...that is... I'm sure thtt that's not 
acceptable. I mean that's just too much. But again it was hard for me as an 
educator to hiow what is too muck You htcw they said, "Tell them to get out 
of their seats. Tell them to go around and look at the boards, you knew and lock 
at all the problems they've done previously." I thought, I don't...htow, I hope 
I'm doing this the right way. I hope that other teachers are doing this, and that 
way it will be fair. But I wasn 't redly sure. 

E7 

On the part I wasn't sure..! don't remember now what the directions said. It 
seemed kind of, ..open as far as it didn't say you have to give your students an 
hour and you must cut it off at that point. So, I don't know...I'm not sure I did 
it wrong by giving them extra time, but with the CIAS test last year I seem to 
remember there was a part on the directions that said give the kids the time that 
they need...so. 

Teachers administered the assessment to their own students. When teachers were uncertain 
of specific administrative procedures they often made their own interpretations. In many 
cases, teachers appeared to take greater license with the administration because of the use of 
classroom materials and the instructional nature of the assessment itself. What is most 
interesting about the variations in administration, is not that they occurred, but that the 
teachers' response was basically to place additional restrictions on the assessment that are 
more consistent with high stakes testing. 



Elevating Hie Stakes of Hie CBAAM 

The Mt. Diablo school system has developed the CBAAM to provide information on student 
achievement using open-ended, constructed response performance tasks. Purposes for the 
CBAAM include: (1) illustrating the types of tasks that are aligned with the Mathematics 
Framework (CDE, 1992), (2) exposing students and teachers to the types of items they will 
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experience on state assessments, and (3) making information available to students, teachers, 
and parents about performance assessment and the concomitant instructional opportunities 
required to enhance student performance. Rigorous standardization in terms of 
administration procedures, elaborate analysis and use of results for accountability are not the 
district s intended purposes. Concerns voiced by teachers about variations in administration 
procedures are consistent with district efforts being directed toward involving teachers in the 
development, field testing, scoring and use of results in their dassrooms-and not indicative 
of an assessment created for accountability and high-stakes use. Much of the variation in 
administration described by teachers is a by-product of the intended purpose of the 
assessment--mainly to enhance student and teacher capacity to engage in complex, yet 
instructionally-sound assessments. Accomplishing this is a long-term, evolutionary process 
that is not easily attained, especially when one considers these assessments to represent 
significant change for participating teachers and students. The district encourages teachers to 
prepare students for the assessment by using instructional materials and strategies related to 
the assessments. In essence, the CBAAM represents a district sanctioned opportunity for 
students and teachers to prepare for higher stakes state assessments that assume district 
alignment with curriculum frameworks. 

Yet, teachers tended to elevate the stakes associated with the assessment in a number of 
ways. In both the survey and interview responses, teachers expressed concerns about the 
legitimacy of allocating instructional time to prepare students for the CBAAM. If preparing 
students for assessment is not appropriate, then engaging in preparation activities is at the 
expense of precious instructional time * high-stakes for students and teachers. 
E5 

Well, I'd say it's interfering to some extent <md that we HA do some prep 
because we didn't want the children to go in cold, especially like with the 
"Carnival" thing. So that took... a long time. It prdbdbhf took dbout a month 
out of our math program to get them ready, <md then. ..it tock..two weeks to give 
it So six weeks...but...I don't feel like that's a waste or mytiung. It's still 
math.. 

E6 

/ did not make it obvious to the kids that it was a test.. .we were preparing for a 
test. Because I wait them know that this is just math <md...this is how I always 
expect them to think their... play out their thoughts...! mean write them down 
and he clear <md all that. 
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E7 

Well, I tried not to because I didn 't want to. 1 didn't want to make it seem 
obvious that we're doing this schedule today because of the CBAAM. I really 
tried ta..blend it with the program because I really didn't want fa../ didn't 
want to make them believe that they have to try hard on this schedule because 
this will be on the test. 

Nil 

Even if somebody gave me the test and showed it to me...two months ahead of 
time. I go back and forth between...do I even want to do that? Is that an 
accurate assessment then? Should I spend the months ahead of time and teach 
them how to fill out a schedule? 

Several non-involved teachers believed that an assessment or test is supposed to illustrate 
what students have learned, but without explicit preparation for that episode. 
N9 

This year C93-'94) was beyond the scope and sequence of the district guidelines 
-tt was not developmentally appropriate and I taught to it specifically. 
Therefore, it is not a measure of children's competence in the areas tested. 

Nil 

[I spent] a week teaching to the test so that my students would be able to do it. 
This makes them feel successful, but make the results meaningless. 

N12 

/ support the idea of the CBAAM, however, the questions must be such that stud 
ents can pull from their prior knowledge and background. The "Carnival 
Problem" was not one of those, and students really needed to be taught many 
lessons for them to have a clue. And this teaching to the test is something I will 
not do and don't believe the purpose of the CBAAM is to assess what students 
know about what I just taught them. This negates the intention of the CBAAM, 
as I understand it. 

N49 

The assessment of children in this manner wouidbe truly meaningful if were 
done in all grades prior to the CBAAM. It is artificial because we had to be 
given sample activities and suggestions to prepare the children for the 
assessment. 

What motivates educators to believe that it is wrong to prepare students for specific targets? 
Is it wrong to teach students exactly what we want them to know and be able to do? 
Teachers have been socialized to believe it is unfair to arm students with specific information 
that enhances their performance on tests. This reflects a concern that preparing students 
destroys the validity of the results, particularly if the results have consequences for students 
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or teachers. This belief is a carryover from what is required to ensure the integrity of the 
results of a high-stakes standardized test. In fact, the tone of some of the comments reflect 
teacher anxiety about walking a fine line between preparation and cheating. An interesting 
complication of this perspective is a concern that students would not be adequately prepared 
for what was a new and complex assessment experience: Many of the teacher comments 
chronicled in the "Impact of the CBAAM" section of this report, illustrate how teacher 
concerns for student readiness caused them to resequence or alter their instruction. Within 
their own classrooms teachers sensed that they could only prepare students to a certain 
point, but the level of preparation tolerance varied significantly among teachers. This 
triggered inter-teacher comparisons that were interpreted as breaches of fairness, because of 
the disparities. 
E6 

The only thing that realty gripes me is that some teachers wrote the answers for 
the "Carnival" on the hoard. They did it. .the whole elms worked together to 
develop a schedule and the kids copied it Then that was scored as if a kid did it. 
So, you can score pretty well if the grown up does it and writes it down for you 
and the kids copy. So that's the only thing that gripes me, cause I know for 
sure my lads, bona fide, earned their score. 

E4 

Even within our school.. I know that there's one teacher that totally prepared 
and really helped them out a lot, ana\..just did a really superb job. I know that 
the students in her class did really well on the test There's another teacher 
that...l could almost promise did not prepare the kids for the test Did not do 
anything above and beyond what that teacher would normally have done. When 
the test came in I happened to be in the room, and the principal let me look them 
over. And he was looking through some of the other tests and he remarked, 
"Wow, this one... you know one class is significantly lower than the other two 
classes." That's all ones and twos. The other two classes are all threes and 
fours and a lot of fives and even a couple of sixes. So I think that.. I think that 
it doesn 't.. always measure what a student's true capabilities are. Because I 
think that we all want the kids to do really well and we want them to feel 
successful And I think Bud maybe.. .even more guidelines for the teachers on 
what they should and shouldn 't be doing would be helpful 

The validity of the CBAAM depends critically on how the information is used. Rather than 
building capacity and moving forward, these comparisons tend to reinforce cultures of 
isolation (Jamentz, 1993). To counter this, teachers and district staff must foster a mileau of 
collaboration and collegiality that serves to improve instructional methods and student 
performance as they relate to complex performance-based tasks. Use of the CBAAM for 
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accountability, or to compare performance at the teacher and school level, speaks against this. 
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VI. Conclusions and Recommendations 



Impact on Third Grade Curriculum and Instruction 

The CBAAM had profound short-term effects on teaching and learning in third grade 
classrooms. Teachers resequenced content, introduced new content, and emphasized 
instructional processes as a direct result of the CBAAM. Even teachers who voiced negative 
feelings about the project made changes. This research documents the changes, but does not 
provide evidence bearing on the quality of the instructional modifications pursued by 
individual teachers. Evidence also suggests that there is a substantial long-term impact, 
primarily related to sanctioning changes in teaching methods and mathematics content 
consistent with ideas forwarded in state frameworks. The CBAAM, in essence, gives teachers 
permission to move away from the structure of traditional mathematics programs. 

The research also reveals that the Mt. Diablo teachers were involved in changes in pedagogy 
long before the CBAAM project. Therefore, teaching tools and methods such as cooperative 
groups, manipulatives, writing and expressing ideas, and using problem-solving activities 
were already integrated into their instructional programs. However, in many cases, the 
CBAAM motivated changes in content fractured the traditional structure and sequence of 
their elementary mathematics programs - and they had little time to make the adjustments. 
Much of the proposed content is new learning for teachers themselves, as well as students. 
Encouraging and sustaining changes in the content of the mathematics programs w ill require 
substantial future support to, and cooperation among, teachers. Professional development 
should place much of the "new" mathematics content outlined in the frameworks in the 
context of sound pedagogical strategies that bring these mathematical concepts to life for 
both students and teachers. 

Use of the CBAAM Results 

Teachers use the CBAAM results to reflect on their own teaching, to articulate content and 
performance standards with students, to encourage students to improve performance, and to 
a lesser extent to communicate with parents. This was in stark contrast to teacher comments 
about standardized test results. Standardized tests served primarily administrative functions 
and high stakes decisions (student placement) while the CBAAM information was used 
primarily in the service of classroom instruction. 
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Professional Development 

Mt. Diablo teachers recognize the current transition period and are struggling to find the 
proper balance. On one hand, they embrace many of the new changes, on the other they 
recognize the merits of some traditional practices. They are exploring personal and 
professional growth in a forest of extremes; text books/no text books, alternative 
assessments /standardized tests, working alone/ working together (students and teachers), 
computation /problem solving, on and on. They see the CBAAM as only one piece in the 
evolving strategy for systemic mathematics reform. However, they are concerned about the 
alignment of state and district guidance with the assessments. They are concerned about 
collaboration and coordination within and across grade levels. They are concerned about 
communication with parents, and they wonder about the time and resources needed to 
alleviate these concerns. They recognize that some combination of formal and informal 
professional development, that is essentially ongoing, will be required to strike the proper 
balance. 

Getting teachers actively involved in the process of within and between grade-level 
articulation may be an important starting poin*. Professional development opportunities 
focuing on essential mathematics content, communicating how that content spirals through 
the years in more complex, yet meaningful ways, will help articuate the ideas in the 
frameworks and translate those ideas into meaningful classroom instruction. It is through 
these efforts that stakeholders including, teachers, administrators, parents, and students, will 
realize that success on district and state assessments in third and fourth grades is dependent 
on a coherent K-4 instructional program. 

Concerns about Assessment Driven Curriculum 

Clearly, the information presented in the Impact of the CBAAM section of this report, 
indicated that third grade curriculum and classroom assessment were altered in direct 
response to the CBAAM. However, some teachers were distressed by having the assessment 
drive the sequence and selection of content in their classroom. When a teacher resists 
spending time on content to be assessed, they risk not providing adequate preparation for 
students in their classes. This could result in poor comparisons with other third grade 
classes. In addition, they are concerned about students' own perceptions of the assessment 
experience and students' satisfaction with their own performance. Instead of viewing the 
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assessment as an opportunity to learn about student performance and improve it subsequent 
to the assessment many teachers interpreted the CBAAM as a summative measure of 
cumulative mastery in the primary grades or mastery of third grade content This 
interpretation substantially elevates the stakes of the assessment. It also diminishes the 
perceived validity of the assessment because the assessment is only three items and covers 
limited content. 

This concern is interesting, given that the district does not use the CBAAM for teacher 
comparisons, student certification, program eligibility, or other high stakes applications. 
Standardization of assessment administration can be critical depending on how the 
assessment results are used. In situations where score interpretations weigh heavily on the 
student, standardization becomes a central concern. In the case of the CBAAM, variations in 
the amount of time spent taking the test, grouping structures or availability of manipulatives 
and other materials may not compromise the integrity of the results. Teachers administer the 
assessment to their own students and use the resulting information in ways they deem 
appropriate to alter instruction or improve student learning. There are no district mandates 
specifying how the results should be interpreted or used by teachers. Thus, in this context, 
teachers play a major role in determining the stakes associated with the use of the CBAAM 
results. There in little doubt that systemwide assessments, whether they are state or district 
level, alter the course of instruction. Mt. Diablo should engage all teachers in discussions of 
the rationale for the current course and the intended effect on students. The essential 
question becomes; is the assessment driving instruction in the right direction? Common 
goals and shared understandings are essential in this period of transition. Beyond common 
goals, clear communication and understanding of intended purpose and resolution of 
uncertainty surrounding administration procedures will be necessary to insure the integrity 
of the assessment results. 

Assessment Consequences 

There are a constellation of consequences associated with the CBAAM. Many of these 
consequences are direct, such as substitution of instructional time with the assessment, 
increasing student opportunity, and placing greater demands on teacher planning. However, 
many consequences are teacher mediated. For example, a few teachers did not return the 
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assessment results to the students, because they did not want students to see their poor 
performance which might result in decreased motivation. These teachers interpreted the low 
scores negatively, rather than seeing them as an opportunity to revise or improve 
performance and enhance learning. Think of this same teacher viewing the assessment as a 
summative measure of the third grade mathematics content and cumulative learning in the 
primary grades, which she had little time to prepare herself and her students for. In 
addition, imagine the results of the assessment being made available to parents and the 
public. Each consequence taken independently does not significantly elevate the stakes of the 
CBAAM. However, when all these consequences are taken together, they can create a 
situation where the CBAAM results weigh heavily on the students and teachers. The 
communication about the purpose and use of the CBAAM needs to be precise and clear to 
teachers and parents. If this clarity is not present, there is a danger of creating an 
environment of resistance and elevated anxiety, that is primarily a function of the lack of 
information. The district should be quite aggressive about advertising the purpose of the 
CBAAM and the intended use of its results. It's not just the district that creates the stakes, 
but often other stakeholders construct a maze of consequences that elaborate and obscure the 
original intent of the assessment. 
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APPENDIX A 



Third Grade Items and Student Booklet from the 1994 
Curriculum-Based Alternatives Assessment of Mathematics 



District-wide Results for Three CBAAM Items 



SO 



Student Name _ 

Student ID 

Teacher Name . 
School Number 



Place label here or fill in blanks 



Curriculum-Based Alternative Assessment 



of 



Mathematics 



Third Grade 



TO THE STUDENT 

Remember that you want to answer each question as clearly and completely as you can. 
Your descriptions and explanations are as important to your score on this test as finding 
an answer or solution. 



NAME 




Ml. Diablo Unified School District 
January 1994 
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« Treasure Island Map 

Part One - Day 1 



Y 



12 
1 1 
10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 



! i : 1 T I I 

! 1 1 ! i 


! 
i 


: i 
1 

1 1 

; 1 


1 ! ! ! 

!-! 










1 ! 

! ! ; 


i 

1 1 


1 

1 






! 1 ! I 

! i ! 


! ; i 








! j ! 


i 


1 | ; 

i ! 
; : ! 




1 

1 

1 

1 






| j 

1 ; 


r 

i 

1 

1 

1 


I 

! 1 


— i 


i — 








i 
1 

i 


1 


! 

i 

> 












— 1 


»— 










i 
i 
i 




1 












i 




1 


t 


i 












\ 1 

1 i i J 




I 

1 

1 


















w 

1 

! 


1 < 


1 — 




















i 

i 
























i 



5 6 

X 



7 8 9 10 11 12 



Key: 1 line = 1 block 
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Treasure Island 



ERIC 



Pan I. In your group: 

1. Complete the Compass Rose by filling in the missing cardinal points. 

N 



-> 



a. Find the Big Willow symbol on the map. Check its coordinate point. Now transfer the symbols 
for Devil's Cove and Wishing Well to the correct coordinate points on the map. 



Big Willow (2,10) 
Wishing Well (9,4) 

Devil's Cove (10 10) 



b. With your group, select common symbols for the next three places. 

Beaver Lake 

Foot Bridge 
Pirate Ship 

c. Put your symbols on the correc coordinate points on the map. 

Beaver Lake (4,6) 
Fool Bridge (1,1) 
Pirate Shh (6,12) 

Create a name and symbol for the remaining three coordinates, and place them on the map. 
(11.6) 

(12,2) 

(6,6) 



STOP! End of Part I 

2 
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Treasure Island 



Pan II. 



While walking home from Treasure Island Elementary School, you noticed a crumpled 
piece of paper sticking out of a crack in a large tree. When you opened the paper, you recognized 
it as directions that appeared to lead to a hidden treasure. The note looked like this: 



Start at the Big Willow, 
Go 3 blocks east 



Go 6 blocks south and 1 block west 
Go 3 blocks west and 2 blocks north 
Go 2 blocks west 

Go 5 blocks south and 3 blocks west 
Dig here for the treasure 



With your pencil and the use of the map, find the hidden treasure by following the directions. 

Write the coordinate point where the treasure is buried: 

Dig for Treasure Here. 
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Treasure Island 

Part III 

Soon after you found the treasure, you heard the pirates were after you. You 
buried the treasure at one of the other symbols on the map. To fool the pirates, you 
chose a route with several turns. 

1. Using a crayon, start at the Big Willow and draw your route on the original map. 
(Hint: Don't make it too easy for the pirates to find the treasure.) 

2. Write a letter to a friend explaining how to find the treasure. 



Date 



Dear 



Your friend, 



4 
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Student Name 

Student ID 

Teacher Name 

School Number Class/Period No. 



Place label here or fill in blanks 



Curriculum-Based Alternative Assessment 

of 

Mathematics 



Third Grade 



4 

TO THE STUDENT 

Remember that you want to answer each question as clearly and completely as you :an. 
Your descriptions and explanations are as important to your score on this test as finding 
an answer or solution. 



NAME 




Ml Diablo Unified School District 
January 1994 



Carnival 

I. The school carnival is coming up. Your class is responsible for the "Wet SDonee 
Toss" booth. 6 

The carnival lasts from 10 a.m. to 2 p.m. 
The jobs are: 

1. taking tickets 

2. passing out and picking up sponges 

3. being the target 

II. Show how you would organize the work schedule for your class's booth. 
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HI. Tell how many srudents you will use to staff the booth and how long each student I 
will work. 



2 
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Hershey Bar 



You have asked three of your friends to come over. Your mother is in a good mood, 
and so she gives you a Hershey Bar to share with your friends. You take the wrapper off 
the Hershey Bar and see that it looks like this: 



| Hershey || Hershey 


Hershey 


Hershey 


Hershey 


1 Hershey | 'Hershey 


Hershey 


Hershey 


Hershey 



Show and tell how you can divide this Hershey Bar so that all four of you each get the 
same amount with no leftovers. 
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199? District Results 



Treasure Island Problem 



Score 


Percent 


6 • Exemplary 


3.5 


5 - Competent 


29.4 


4 - Satisfactory 


19.6 


3 - Nearly 
Satisfactory 


34.9 


2 - Flawed 


10.9 


1 - Very Poor 


1.8 


Carnival Problem 


Score 


Percent 


6 * Exemplary 


0.2 


5 - Competent 


7.5 


4 - Satisfactory 


24,5 


3 - Nearly 
Satisfactory 


30.1 


2 - Flawed 


30.6 


1 - Very Poor 


7.1 


Hershey Bar Problem 


Score 


Percent 


6 - Exemplary 


1.1 


5 - Competent 


13.1 


4 - Satisfactory 


34.3 


3 - Nearly 


32.1 


2 - Flawed 


14.9 


1 - Very Poor 


4.4 
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Third Grade Survey and Cover Letters 
Interview Guiding Questions 
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FarWest 



LABORATORY 



Dear Teacher, January 30, 1994 

The Mt. Diablo Curriculum-Based Alternative Assessment of Mathematics (CBAAM) 
is one of many local assessment projects supported and studied by the California 
Assessment Collaborative (CAC) and the Far West Laboratory. The CAC and Far 
West Laboratory are working with Mt. Diablo staff to document and understand the 
impact of CBAAM implementation on teaching and learning. For several years, staff 
from the Far West Laboratory have participated with teachers in planning and 
development sessions. To better understand the impact of CBAAM implementation, 
we would like to hear from teachers beyond the development teams. In order to do 
this, we have received permission from the Mt. Diablo School District for third and 
sixth grade teachers to complete a short survey and, if interested, participate in a 
follow-up interview. 

The enclosed survey should take about 20 minutes to comple f e. We are requesting 
all third and sixth grade faculty who teach mathematics to complete the survey. The 
first page asks for background information and the next five pages ask you to use a 
rating scale to respond to statements. We value your input and welcome additional 
comments. Please use the space provided at the end of the survey for this purpose. 

The final page of the survey invites you to participate in a follow-up interview in late 
February. Interviews will be conducted at your school site at the end of the school 
day. Far West Laboratory' will pay a $40.00 honorarium to each teacher involved in 
an interview. The interviews should take about 60-90 minutes. We are interested in 
hearing from teachers with different backgrounds and different levels of participation 
in the CBAAM development. Approximately 20 teachers will be selected for 
individual interviews. 

Please return the survey and interview request to Far West Laboratory by February 
18th. We have enclosed a stamped, self-addressed envelope for mailing. A summary 
of the survey results will be returned to all third and sixth grade teachers. Your 
responses will be kept confidential and in no case will data be linked to individual 
teachers or schools. 

Thank you very much for your time. 

Sincerely, 



Peter Ryan 
Research Associate 
Far West Laboratory 

For W«it laboratory for Educotionol Research and Development 

"30 Hofft$on Street San ffonciuo CA 94107 1242 (415) 565-3000 *AX (415) 565-3012 



MT. DIABLO 

CURRICULUM-EASED ALTERNATIVE ASSESSMENT OF MATHEMATICS SURVEY 



Name: ; School: 

(optional) 

Phone: ( ) Grade Level: Years Teaching: 

(optional) 

Content Areas Taught: Number of Years: Content Area* Taught: Number of Years: 

All subjects (Elementary) History/Social Science 

English-Language Arts Visual^erforming Arts 

Math Foreign Language 

Science ESL 

Other 



Subject Matter Preparation in Mathematics: (Check all that apply) 

Minor(BA/BS) in Mathematics 

Major(BA/BS) in Mathematics 

Advanced Degree in Mathematics Degree Earned 

Related Degrees (Please describe) 



Professional Development other than Formal Degrees (e.g., subject matter projects, EQUALS) 



Involvement in Mt Diablo Curriculum-Based Alternative Assessment of Mathematics (CBAAM). 
Please check all thai apply: 

91-92 92-93 93-94 

I was not involved in any CBAAM development, field testing, or scoring. 

1 was involved in assessment item development. 

I was involved in piloting or field testing items. 

1 participated in scoring CBAAM student work at the district level. 

q Ml bublo Curriculum <B*m4 Survey Far Wctt Ubontory 1.94 

ERIC 93 



A. Student Experiences 



Please use the scale below to estimate the frequency 
of student participation in the following activities 
in your mathematics classes. 



2x 



BlOfc Monthly Monthly Weekly Daily 



1. Students write about their mathematical thinking 
to support and clarify their solutions. 

2. Students work together on problems, sharing and 
discussing ideas, strategies, and solutions. 

3. Students use tools such as manipulative*, 
calculators, computers, and rulers. 

4. Problems and instructional activities require students 
to apply mathematical ideas to the world beyond the 
classroom. 

5. Students use as visual representations, 
computational algorithms, and models to 
to illustrate math thinking. 

6. Students work independently to complete problems 
focusing on mastery of specific computational skills. 

7. Students are exposed to complex open-ended tasks 
that have multiple solutions. 

8. Students complete tasks that involve investigating 
mathematical ideas over extended periods of time. 
Problems and related activities may involve several 
class periods over several days. 

9. Students reflect on their work (orally or in writing) 
to express new learning or understandings. 

10. Homework involves formulating, completing, or 
revising open-ended tasks. 

11. Students use scoring rubrics or scoring criteria to 
evaluate their own work. 

12. Students revisit and revise their solutions/responses 
to open-ended problems. 

13. Mathematics is integrated into instructional activities 
in other sutytct areas. 

14. Homework involves problem sets that reinforce 
basic skills. 



(2) 
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Please refer to the scale below to indicate the 
amount of student exposure to the following 
math content 

One Two More Than Complete Unitls) 
Never LfiJfiD lessons 2 LCMM MM1Y Itffiflf* 



15. Students represent and compare quantities using 
manipulative and visual representations. 

16. Students collect organize, and represent data 
in charts, tables or graphs. They write about 
their analysis and interpretation of data. 

17. Classroom activities deal with understanding and 
using coordinates and coordinate systems. 

18. Classroom activities focus on part-to-whole 
relationships and visual representations of 
fractions. 

19. Students create and rtad maps (for example, they 
use directions, locate landmarks, estimate distance, 
and interpret legends). 

20. Students create and intrrpret schedules involving 
time and other dimensions. 



(3) 



9f) 



HI th*Uo CumeuJum-BiMi Survty 



Fir W«t Ufconiory 1 94 



B. Teacher Experiences/Opinion 



Please read the following statements and circle the response 
that best indicates your view. 

Strongly Partially Strongly 

Ctasof dames abb Mat Agree 



1. It is important to meet with a group or 
turn of teachers regularly to discuss 
mathematics content and instructional 
strategies. 

2. In my claw, I hive altered the way students 
"do" mathematics to correspond to the 1992 
Mathematics Framework. (For example, 
collaborative work, writing about math, and 
extended tasks or investigations). 

3. I have ample opportunity to review and discuss 
examples of problems and student work related to 
the content and standards in the 1992 
Mathematics Framework. 



1 2 



1 2 



1 2 



in the past two yean, the district has provided 
professional development opportunities that 
enhanced my understanding and teaching of 
mathematics. 



1 2 



I have altered the content of my instructional 
units to correspond to content covered by the 
CBAAM. 



1 2 



6. There is adequate time to teach basic 

computational skills and more complex problem 

solving. 



1 2 



7. Textbooks art the primary source of instructional 
guidance and classroom activities. 



1 2 



8. In my class, I have altered the way students 
"do 9 mathematics to correspond to the tasks on 
the CBAAM. 



1 2 



Classroom activities are often based on the 
application of mathematics to real world 
situations. 



1 2 



10. I am confident in my ability to develop and 
use complex problems and investigations in 
my teaching. 



1 2 



(4) 
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Strongly Partially Strongly 

QiltitQi Qiaffltt ABB! Aurtt Aflrtt 



11. Teachers were clearly informed of the general 
content and format of tasks on the CBAAM. 


1 2 


3 


4 5 


12. I think recent changes in mathematics 
instruction and assessment are necessary 
and important 


1 2 


3 


4 3 


13. 1 have altered the content of my instructional 
units to correspond to content described in the 
1992 Mathematics Framework. 


1 2 


3 


4 5 


14. Feedback about student performance on the 
CBAAM will be valuable in my future 
instructional planning. 


1 2 


3 


4 3 


IS. Prior to CBAAM administration, 1 reviewed 
the generic scoring rubric with my students. 


1 2 


3 


4 5 


16. Tasks on the CBAAM mesh well with the type 
of problems students are exposed to in my 
math classes. 


1 2 


3 


4 3 


17. In recent years, 1 have been involved in 

valuable professional development experiences 
(in math) outside of the district 


1 2 


3 


4 5 


18. Students will benefit by reviewing and 

understanding their performance on the CBAAM. 


1 2 


3 


4 5 


19. My students were well prepared for the content 
and task format on the CBAAM. 


1 2 


3 


4 5 


20. The time and effort spent developing and 
scoring the CBAAM is worthwhile in terms of 
the benefits to teachers and students. 


1 2 


3 


4 5 


21. The district should only use assessment results 
to provide feedback to individual students and 
teachers. 


1 2 


3 


4 5 


22. Administration of the CBAAM and similar 
statewide assessments will motivate teachers to 
change instructional practices. 


1 2 


3 


4 5 


23. 1 would like to have more systematic interaction 


1 2 


3 


4 5 



with other teachers that are developing and 
using alternative assessments. 
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C. Classroom Assessment 



Please use the scale to indicate how often 
you use the assessment tools listed below. 



KevcT BiBlS Monthly Weekly Qlfc 



1. Open-ended tasks 

2. Portfolio 

3. Teacher-made tests and quizzes 

4. Textbook tests and quizzes 

5. Developmental scales 

6. Investigations 

7. Projects 

8. Exhibitions 

9. Anecdotal records 

10. Holistic or analytic scoring rubrics 

11. Operational checklists 

11 Student self-assessment or reflection 

13. Oral questions/interviews 

14. Attitude surveys 

15. Peer assessment 



Please use this space to provide additional comments abou 
assessment project, and what you think the project has or has not accomplishe« 




5 
5 
5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 
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Interview Request 



Far West Laboratory would like to interview third and sixth grade teachers in an 
effort to obtain more detailed information about the relationship among current 
instructional practices in mathematics, teacher professional development 
opportunities, and the CBAAM assessment project. Interviews will take place in 
February and be conducted after classes at school sites. Interview times will be 
arranged in consultation with participating teachers. Interviews will take 60-90 
minutes and Far West Laboratory will pay a $40.00 honorarium to each participant. 
Teachers will be selected based on differences in background and CBAAM 
development history. Therefore, some interested teachers may not be contacted for 
interviews. 

The interviews will be audiotaped and transcribed. Following transcription 
recordings will be destroyed. The source of all data collec f ed will be coded to insure 
confidentiality. Only the Far West Laboratory CAC project staff will have access to 
the data. In no case will data or the dissemination of study results be linked to actual 
teachers, or schools. Upon completion, study findings will be shared with teachers 
and district staff. 

Please Check one of the following to indicate your interest: 



*Yes, I would li' . to participate in a follow-up interview. Far West 
Laboratory Staff can contact me a the following number(s): 

Name Phone 



No, I am not interested in further participation. 



If you have questions about the survey or interviews, please feel free to call Peter 
Ryan at Far West Laboratory (415-565-3031). 
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Guiding Questions: Interview 



Many teachers feel that the CBAAM i; having an impact on how they teach mathematics while 
others don't see an impact. What is y««r experience in your classroom and school? 

Were you satisfied with student performance on the assessment? How did you use the results in 
your classroom? What was the students reaction? 

In your opinion is the assessment helping teachers and students move mathematics instruction and 
performance in the direction of the state frameworks? Why or why not? 

What do you think needs to happen in order to enhance teacher and student capacity to successfully 
engage in tasks /activities like those on the CBAAM? What kind of staff development and student 
experience is necessary? 

What part of your mathematics program does the CBAAM measure particularly well? What do you 
teach that the assessment doesn't measure? What changes would you like to see? 
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